2
$\begingroup$

Question: is minimizing test set mean validation error more important than the gap between train and test errors?

Let's say I can tweak parameters in my model to give me mean validation error of 4500 RMSE on k-fold cross validation. When I use these parameters I found to compute RMSE on train and test, I get 1500 and 4500, respectively - quite overfit.

Is that still more preferable than a model that would get me, say, 5000 train error, and 5500 test error? Something that performs worse on the test set, but is less overfit to the training set?

Improve this question
$\endgroup$
1
  • $\begingroup$ I don't know where I once read (probably here on SE or a tutorial): once you get tour model running don't even bother looking at the train error (or other evaluation metrics), it's not much informative. $\endgroup$
    – Firebug
    Commented May 30, 2016 at 21:15

1 Answer 1

Reset to default
3
$\begingroup$

Generally, yes. In a context where you're trying to estimate test error, what you probably care about is how well the model will predict unseen values. Test error is what tells you this, not the gap between test error and training error. A model that heavily overfits but has better test error than another model is still more likely to be more accurate for future observations.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Sorry for getting back after that much of time but do you maybe have any reference regarding this statement? $\endgroup$
    – TheDude
    Commented Jul 15, 2019 at 8:10
  • $\begingroup$ @TheDude What exactly do you think calls for a reference? The idea that test error tells you how well the model predicts unseen values? $\endgroup$
    – Kodiologist
    Commented Jul 15, 2019 at 12:38
  • $\begingroup$ I mean a reference for the statement that you would basically prefer an heavily overfitted model with a better test error than a less overfitted error that might not have the same performance on the test set as the former model. Is there a reference where this is pointed out somehow as a rule of thumb? The reason why I'm asking that I'm basically running into this trade-off for my thesis' models and I have to somewhat show proof why I would go with either option. $\endgroup$
    – TheDude
    Commented Jul 15, 2019 at 13:20
  • $\begingroup$ @TheDude No, I don't remember seeing it written elsewhere. In order to demonstrate this mathematically, you would have to look at how well the train–test gap estimates generalization error (so you could compare it to test error), which I don't think anybody's done because there's no obvious reason to do it. By contrast, the degree to which test error estimates generalization error is a well-studied topic; see for example section 7.12, "Conditional or Expected Test Error?", in Hastie, Tibshirani, and Friedman (2009; web.stanford.edu/~hastie/ElemStatLearn ). $\endgroup$
    – Kodiologist
    Commented Jul 15, 2019 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.