Suppose you have a training and testing set. You fit two models, A and B, to the training set. They you predict on the testing set. You find (in this contrived example):

Test MSE model A: 3 Test MSE model B: 4

It seems like in practice I don't see people bootstrap test MSE (i.e., get a bootstrap sample from the training set, fit models A and B, predict on test set using refitted models, repeat 100 times or so), but it seems like a valid thing to do.

My question is, from a statistical inference perspective, is this something we can/should do? In other words, would a statistician say "3 and 4? These numbers are meaningless without intervals. Bootstrap the test MSE and do a statistical test."

On the other had, I could see an objection: we are trying to assess out-of-sample performance, so why would we use the training set to make inferential statements?

Also, if you could include a source for further reading, that would be ideal.


In an "Introduction to Statistical Learning" by Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani, in the section entitled "Validation and Cross-Validation," they promote calculating the standard error of the test MSE as a valid way to compare models. However, it should not be done through bootstrapping. Rather, one should randomly split the testing and training many times, recalculating the test MSE with each iteration.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.