Question: is minimizing test set mean validation error more important than the gap between train and test errors?

Let's say I can tweak parameters in my model to give me mean validation error of 4500 RMSE on k-fold cross validation. When I use these parameters I found to compute RMSE on train and test, I get 1500 and 4500, respectively - quite overfit.

Is that still more preferable than a model that would get me, say, 5000 train error, and 5500 test error? Something that performs worse on the test set, but is less overfit to the training set?

  • $\begingroup$ I don't know where I once read (probably here on SE or a tutorial): once you get tour model running don't even bother looking at the train error (or other evaluation metrics), it's not much informative. $\endgroup$ – Firebug May 30 '16 at 21:15

Generally, yes. In a context where you're trying to estimate test error, what you probably care about is how well the model will predict unseen values. Test error is what tells you this, not the gap between test error and training error. A model that heavily overfits but has better test error than another model is still more likely to be more accurate for future observations.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.