Suppose you have a training and testing set. You fit two models, A and B, to the training set. They you predict on the testing set. You find (in this contrived example):
Test MSE model A: 3 Test MSE model B: 4
It seems like in practice I don't see people bootstrap test MSE (i.e., get a bootstrap sample from the training set, fit models A and B, predict on test set using refitted models, repeat 100 times or so), but it seems like a valid thing to do.
My question is, from a statistical inference perspective, is this something we can/should do? In other words, would a statistician say "3 and 4? These numbers are meaningless without intervals. Bootstrap the test MSE and do a statistical test."
On the other had, I could see an objection: we are trying to assess out-of-sample performance, so why would we use the training set to make inferential statements?
Also, if you could include a source for further reading, that would be ideal.