1
$\begingroup$

Question: is minimizing test set mean validation error more important than the gap between train and test errors?

Let's say I can tweak parameters in my model to give me mean validation error of 4500 RMSE on k-fold cross validation. When I use these parameters I found to compute RMSE on train and test, I get 1500 and 4500, respectively - quite overfit.

Is that still more preferable than a model that would get me, say, 5000 train error, and 5500 test error? Something that performs worse on the test set, but is less overfit to the training set?

$\endgroup$
  • $\begingroup$ I don't know where I once read (probably here on SE or a tutorial): once you get tour model running don't even bother looking at the train error (or other evaluation metrics), it's not much informative. $\endgroup$ – Firebug May 30 '16 at 21:15
3
$\begingroup$

Generally, yes. In a context where you're trying to estimate test error, what you probably care about is how well the model will predict unseen values. Test error is what tells you this, not the gap between test error and training error. A model that heavily overfits but has better test error than another model is still more likely to be more accurate for future observations.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Sorry for getting back after that much of time but do you maybe have any reference regarding this statement? $\endgroup$ – TheDude Jul 15 '19 at 8:10
  • $\begingroup$ @TheDude What exactly do you think calls for a reference? The idea that test error tells you how well the model predicts unseen values? $\endgroup$ – Kodiologist Jul 15 '19 at 12:38
  • $\begingroup$ I mean a reference for the statement that you would basically prefer an heavily overfitted model with a better test error than a less overfitted error that might not have the same performance on the test set as the former model. Is there a reference where this is pointed out somehow as a rule of thumb? The reason why I'm asking that I'm basically running into this trade-off for my thesis' models and I have to somewhat show proof why I would go with either option. $\endgroup$ – TheDude Jul 15 '19 at 13:20
  • $\begingroup$ @TheDude No, I don't remember seeing it written elsewhere. In order to demonstrate this mathematically, you would have to look at how well the train–test gap estimates generalization error (so you could compare it to test error), which I don't think anybody's done because there's no obvious reason to do it. By contrast, the degree to which test error estimates generalization error is a well-studied topic; see for example section 7.12, "Conditional or Expected Test Error?", in Hastie, Tibshirani, and Friedman (2009; web.stanford.edu/~hastie/ElemStatLearn ). $\endgroup$ – Kodiologist Jul 15 '19 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.