Why do we worry about overfitting even if "all models are wrong"?

Question

I am wondering why we care so much about overfitting. We know that statistical models are tools to tell us some information, but they are not even fully accurate.

Answer 1

The quote by Box is along the lines of "All models are wrong, but some are useful."

If we have bad overfitting, our model will not be useful in making predictions on new data.

Answer 2

Why do we worry about overfitting even if “all models are wrong”?

Your question appears to be a variation of the Nirvana fallacy, implicitly suggesting that if there is no perfect model, then every model is equally satisfactory (and therefore flaws in models are irrelevant). Observe that you could just as easily ask this same question about any flaw in a model:

• Why do we worry about maximum likelihood estimation even if “all models are wrong”?

• Why do we worry about standard errors even if “all models are wrong”?

• Why do we worry about cleaning our data even if “all models are wrong”?

• Why do we worry about correct arithmetic even if “all models are wrong”?

The correct answer to all such questions is that we should not make the perfect the enemy of the good --- even if "all models are wrong", a model that is less wrong is still preferable to a model that is more wrong.

Answer 3

The full quote is "All models are wrong, but some are useful". We care about overfitting, because we still want our models to be useful.

If you are familiar with the Bias-variance tradeoff, the "all models are wrong" statement is roughly equivalent to saying "all models have non-zero bias". Overfitting is the issue that while we can increase the number of parameters in a model to reduce the bias, typically the more parameters we have, the more variance there will be in our estimate. A useful model is one that balances between being flexible enough to reduce the bias, but not so flexible that the variance is too high.

Answer 4

A Citroën 2CV is, in many respects, a poor car. Slow, unrefined and cheap. But it is versatile and can operate effectively on both paved road and freshly ploughed fields.

An F1 car by comparison, is seen as the pinnacle of automotive engineering. Fast, precise and using only the finest components. I wouldn't fancy driving one across an open field though.

The 2CV has general applicability, while the F1 car only has very specific applicability. The F1 car has been overfitted to the specific problem of going round a racetrack as quickly as possible with the benefit of a team of professional engineers to monitor, assess and problem solve any issues that may arise from high performance operation.

Similarly, an overfitted model will perform well in situations it is overfit, but poorly (or not at all) elsewhere. A model with general applicaility will be more useful if it will be exposed to different environments out of your control even if it is not as good as specific models.

Answer 5

As others have noted, the full quote is "all models are wrong, but some are useful."

When we overfit a data set, we create a model that is not useful. For instance, let's make up some data: set.seed(123)

x1 <- rnorm(6)
x2 <- rnorm(6)
x3 <- rnorm(6)
x4 <- rnorm(6)
y <- rnorm(6)

which creates 5 variables, each a standard normal, each with N = 6.

Now, let's fit a model:

overfit <- lm(y~x1+x2+x3+x4)

The model has $R^2$ of 0.996. x2 has a significant p value and x4 is almost sig. (at the usual level of 0.05).

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept) -0.54317    0.08887  -6.112   0.1032  
x1           2.01199    0.14595  13.785   0.0461 *
x2           0.14325    0.08022   1.786   0.3250  
x3           0.45653    0.08997   5.074   0.1239  
x4           1.21557    0.15086   8.058   0.0786 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.1601 on 1 degrees of freedom
Multiple R-squared:  0.9961,    Adjusted R-squared:  0.9805 
F-statistic: 64.01 on 4 and 1 DF,  p-value: 0.09344

It fits the data almost perfectly e.g. try

plot(predict(overfit),y)

But it's all random noise.

If we try to apply this model to other data, we will get junk.

Answer 6

Every model has an error. The best model is that which minimizes the error associated with its predictions. This is why models are typically constructed using only a proportion of the data (in-sample), and then applied to the remaining 'out of sample' data set. An over-fitted model will typically have a greater prediction error in practice than a well formulated one. In addition, a model should be intellectually robust: there is no point constructing a model that works in one 'regime' if it does not work at all in the event of regime change. Such a model might appear to be very well formed until such time as the regime changes because essentially such a model has been constructed 'in-sample'. Another way of saying that is that the model's expected error must be well formulated too. There is also the matter of 'Occam's Razor', which is a philosophical idea that essentially the model should be the simplest possible, using the least number of variables required to describe the system being modelled. This serves as a useful guide, rather than a set-in-stone rule, but I believe that this is the idea behind using the 'adjusted R squared' rather than the R squared, to adjust for the natural improvement in fit associated with using more variables (e.g. you would have perfect fit, an R squared of 100% if you had a separate variable for every piece of data!). It is also an idea that should be applied to modern ML techniques: throwing e.g. thousands of variables at an ML algorithm is dangerous unless you have millions of pieces of data (and even then ... you might be better off transforming your data to reduce the number of variables first). One final point: every model requires belief. Even our laws of Physics are based on observation, and indeed they have required modification as we moved from Newtonian physics into the realms of the very small (Quantum mechanics) and the very large (General Relativity). We cannot say with absolute certainty that our current laws of Physics will hold in the future, or even in the past (e.g. around the time of the big bang). But appealing to our philosophical belief in Occam's razor results in us accepting these models and ideas because they are the simplest models yet devised that fit our observations and data.

In summary, there are no hard and fast rules. Imagine a complex (chaotic?) dynamical system, for example, the global economy. You might construct a well-formed model that works well for a short period of time. But 'regime change' is a very real issue: the economic system is highly complex and non-linear, and there are far more variables than you can measure, that might be of no consequence in the in-sample regime, but of huge significance in another 'regime'. But within your short, essentially in-sample, period, you might find that linear regression works quite well. Common sense should prevail: sometimes a very complex model is required, but it should be heavily caveated if the error associated with its predictions is unknown.

I'm sure that a proper statistician can give a much better answer than this, but since none of the above points seem to have been made yet, I thought that I would stick my neck out ...

Answer 7

All models are wrong, but some are less wrong than others.

Overfitting generally makes your model more wrong in dealing with real-world data.

If a doctor were to try to diagnose whether you have cancer, would you rather have them be wrong 50% of the time (very wrong) or 0.1% of the time (much less wrong)?

Or, let's say you give away something for free if your model predicts this will lead to the customer buying something later. Would you rather give away many things for free without this making a difference to whether customers buy things later (quite wrong) or have most customers come back to buy things later (less wrong)?

Clearly less wrong is better.