2
$\begingroup$

Suppose we train two models on a training set, and then test them both on the training set itself, and on a test set. We have some accuracy metric we're using to evaluate them.

Both models score equally on the test set. However, one model scores higher on the training set than the other. (Assume they both score higher on the training set than on the test set.) Without knowing anything else, is the model whose training and test results are closer together a better model? It seems like it is overfitting less.

If we're looking only to have the model that generalizes best when predicting new data, it seems like both should be considered equally good. Yet, my intuition tells me that I prefer the one whose train and test scores are closer together, though I can't confidently think of a convincing reason for this. Is there any reason to believe that one is better than the other?

$\endgroup$
1
  • $\begingroup$ Some good reading on generalization, which is actually being re-thought recently due to the success of deep learning: arxiv.org/pdf/1710.05468.pdf $\endgroup$ Jan 5, 2019 at 19:48

1 Answer 1

0
$\begingroup$

It's artificial to think these would be the only two choices, but let's stay with your question.

In actual practice, I would choose the model in which training and test scores were closer together, particularly if there was model interpretation to be done.

If training and test are far apart, there's something wrong with the model, often that it's more complicated than necessary (e.g. decision tree has more nodes than it needs). So, you can make an Occam's razor case for the model in which training and test perform similarly, because the one model has unnecessary stuff.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.