Misconception about overfitting

Question

I hear countless times at my job that the gap between train and test implies that there's overfitting. Some people go as far as saying that the goal of model selection is to reduce the gap between train and test. The same people also say that a model with a lower train and test gap but worse test performance is a model that generalizes better.

To me, this seems like a profound misunderstanding about what overfitting and generalization means, and also the bias-variance trade-off.

I took this graph from Elements of Statistical Learning. We see that in this image as we vary complexity, the best fit model is one where the gap between train and test is relatively high, at least with respect to all the models to the left. If we were to do what my coworkers suggest, which is to decrease the gap between train and test, we'd almost always be selecting significantly underfit models.

Am I crazy? Are my coworkers right? Please, I need resolution.

Answer 1

I hear countless times at my job that the gap between train and test implies that there's overfitting.

This is true, but that does not mean the model is a bad one.

Some people go as far as saying that the goal of model selection is to reduce the gap between train and test. The same people also say that a model with a lower train and test gap but worse test performance is a model that generalizes better.

I don't understand the rationale here. Why would I use a model which has a larger generalization error than an alternative model? The advice not to over fit is good, but that does not mean that all over fitting is bad. I mean, the idea that there is some distinct line in the sand beyond which we can say we have overfit is dubious. There are degrees of overfitting, and you have to determine if the amount you have is acceptable.

Even very simple models overfit. Here is an example of that happening

library(rms)

x = rnorm(100)
y = 2*x + 1 + rnorm(100, 0, 3)
model = ols(y~x, x=T, y=T)
validate(model)

          index.orig training    test optimism index.corrected  n
R-square      0.2493   0.2640  0.2365   0.0274          0.2219 40
MSE          10.4684  10.0754 10.6462  -0.5707         11.0391 40
g             2.1228   2.1498  2.1228   0.0270          2.0958 40
Intercept     0.0000   0.0000  0.0238  -0.0238          0.0238 40
Slope         1.0000   1.0000  0.9942   0.0058          0.9942 40

This is possibly the best scenario you can find yourself in. I've got the likelihood and the functional relationship correct, and what do we see? The optimism in all metrics is non-zero. Has the model overfit? Yes, even though it is the "right" model. Does this mean this model is bad? It depends, overfitting is a spectrum and not a dichotomy.

Anyway, let me offer a definitive answer. All models overfit to a degree. That I get a different model when I train on a different dataset means that any given model will have some performance degradation when used on new data. The size of that degradation is a function of sample size and model complexity, but the model can have performance degradation and still be a good model (cue the Box quote). It is the job of the analyst to estimate the degradation and decide if the model is still sufficiently good at its job in order to be used. If your coworkers are purposfully choosing models simply because they have 0 degradation between train and test, even when models with superior generalization error are available, I would probably stop listening to what they have to say quite frankly.