# Why do we worry about overfitting even if “all models are wrong”?

I am wondering why we care so much about overfitting. We know that statistical models are tools to tell us some information, but they are not even fully accurate.

• No suit of clothes has exactly the shape of its wearer. Why, then, should we care about good tailoring? – whuber Oct 31 '19 at 13:57
• "All" models really means all models. Including the model of your surroundings that your brain forms when interpreting the light signals it receives. So just close your eyes, you can't trust them anyway. – Bridgeburners Oct 31 '19 at 15:04
• > [...] but they are not even fully accurate. They don't have to be fully accurate to be useful. – Laksan Nathan Oct 31 '19 at 15:09
• -1. There is plenty of literature on this subject, much of it easily accessible to non-experts (such as myself). It's not clear what the question is, or that there's even a question at all. – Josh Nov 1 '19 at 15:53
• All models are wrong, but some are a lot less wrong than others. – qwr Nov 1 '19 at 17:40

The quote by Box is along the lines of "All models are wrong, but some are useful."

If we have bad overfitting, our model will not be useful in making predictions on new data.

• It really should be "All models are wrong, but some are more useful than others." – Acccumulation Oct 31 '19 at 23:05
• Currently, this is the answer with the most upvotes. I think this question deserves a longer answer, e.g. one that explains why overfitting implies poor generalization, what causes overfitting and maybe even what overfitting is and what it is not. – Dirk Nov 1 '19 at 5:59
• For example: If overfitting would be defined as getting a sample prediction error that is way below the error in the data, then I do not see how or why this implies poor generalization. – Dirk Nov 1 '19 at 6:01

Why do we worry about overfitting even if “all models are wrong”?

Your question appears to be a variation of the Nirvana fallacy, implicitly suggesting that if there is no perfect model, then every model is equally satisfactory (and therefore flaws in models are irrelevant). Observe that you could just as easily ask this same question about any flaw in a model:

• Why do we worry about maximum likelihood estimation even if “all models are wrong”?

• Why do we worry about standard errors even if “all models are wrong”?

• Why do we worry about cleaning our data even if “all models are wrong”?

• Why do we worry about correct arithmetic even if “all models are wrong”?

The correct answer to all such questions is that we should not make the perfect the enemy of the good --- even if "all models are wrong", a model that is less wrong is still preferable to a model that is more wrong.

The full quote is "All models are wrong, but some are useful". We care about overfitting, because we still want our models to be useful.

If you are familiar with the Bias-variance tradeoff, the "all models are wrong" statement is roughly equivalent to saying "all models have non-zero bias". Overfitting is the issue that while we can increase the number of parameters in a model to reduce the bias, typically the more parameters we have, the more variance there will be in our estimate. A useful model is one that balances between being flexible enough to reduce the bias, but not so flexible that the variance is too high.

• @CagdasOzgenc, this is interesting. How exactly do you define model bias? (A trivial example supplementing the theoretical answer could be helpful.) A related thread is What is the random variable when we talk about high variance model or high bias model?. Your answer there could be appreciated as well; I am still struggling with the formulation of mine. – Richard Hardy Nov 1 '19 at 14:50
• "Standard literature assumes that the parametric model subsumes the truth hence we can talk about a bias vector approaching zero." The point of the "All models are wrong" quote is that, while we often start with this assumption because it makes the math easier, it is a fundamentally flawed assumption. – Cliff AB Nov 1 '19 at 17:17
• @CagdasOzgenc, is your definition of model bias in line witht the following characterization? Expected squared prediction error can be additively decomposed into squared bias, variance and irreducible error. The deterministic part of the model results in expected squared error equal to squared bias + variance. Under perfect estimation precision, the variance is zero. Hence, squared bias is the expected squared error in estimating the deterministic part of the outcome when estimation precision is perfect. Thus bias reflects the best possible approximation of the DGP allowable by the model. – Richard Hardy Nov 1 '19 at 19:47
• @CagdasOzgenc: I think your argument then is "there are models that are not wrong". That is worth discussing, but it gets considerably more complicated. Also, we typically do introduce bias to such models without limiting the flexibility (i.e., regularization) in order to tame the variance. There's several different ways one could go discussing this, so I'd say it's beyond the scope of this question. – Cliff AB Nov 1 '19 at 20:36
• @CliffAB, regularization introduces bias in a parameter estimator, not in a model. For example, if the model (the functional form) happens to coincide with the DGP (however unlikely, let us assume that for the sake of the illustration), there is no model bias. However, we can still introduce bias in a parameter estimator of that model by doing some regularization. – Richard Hardy Nov 1 '19 at 20:40

A Citroën 2CV is, in many respects, a poor car. Slow, unrefined and cheap. But it is versatile and can operate effectively on both paved road and freshly ploughed fields.

An F1 car by comparison, is seen as the pinnacle of automotive engineering. Fast, precise and using only the finest components. I wouldn't fancy driving one across an open field though.

The 2CV has general applicability, while the F1 car only has very specific applicability. The F1 car has been overfitted to the specific problem of going round a racetrack as quickly as possible with the benefit of a team of professional engineers to monitor, assess and problem solve any issues that may arise from high performance operation.

Similarly, an overfitted model will perform well in situations it is overfit, but poorly (or not at all) elsewhere. A model with general applicaility will be more useful if it will be exposed to different environments out of your control even if it is not as good as specific models.

• Devil's advocate: The problem with driving an F1 over an open field is not that the F1 is such a good fit for race tracks, (I could imagine that there may be a car which is a great fit for both open fields and racetracks), but that the F1 is just a bad fit for open fields. – Dirk Nov 1 '19 at 6:24
• @Dirk actually yes, quite literally, the problem with driving an F1 over an open field is that it's such a good fit for race tracks. Namely, it fits very well to the ground of a flat raceway (low ground clearance), but therefore isn't as flexible to also fit to anything non-flat. A normal car has more flexible suspension, which means it doesn't “stick to the pavement” as well but in return manages also some other tasks. — “There may be a car which is a great fit for both open fields and racetracks” – that would require very good active suspension, probably be heavy and therefore slower. – leftaroundabout Nov 1 '19 at 9:52
• I don't find this a very good analogy. A severely overfitted model (such as an n-degree polynomial fitted to n+1 points) isn't useful for anything. A F1 is not overfitted, it is just a highly specialised tool useful for a very specific role. The statistical analogy would be a model that is trained and useful for a very specific type of forecasting, but not useful for other roles; such a model is not overfitted, just very limited in scope. – gerrit Nov 1 '19 at 12:13
• @gerrit the overfitted model predicts n+1 points exactly. It's only useless elsewhere. – Caleth Nov 1 '19 at 12:26
• @Caleth The training points are not a forecast/prediction, they are a measurement. – gerrit Nov 1 '19 at 12:29

As others have noted, the full quote is "all models are wrong, but some are useful."

When we overfit a data set, we create a model that is not useful. For instance, let's make up some data: set.seed(123)

x1 <- rnorm(6)
x2 <- rnorm(6)
x3 <- rnorm(6)
x4 <- rnorm(6)
y <- rnorm(6)


which creates 5 variables, each a standard normal, each with N = 6.

Now, let's fit a model:

overfit <- lm(y~x1+x2+x3+x4)


The model has $$R^2$$ of 0.996. x2 has a significant p value and x4 is almost sig. (at the usual level of 0.05).

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.54317    0.08887  -6.112   0.1032
x1           2.01199    0.14595  13.785   0.0461 *
x2           0.14325    0.08022   1.786   0.3250
x3           0.45653    0.08997   5.074   0.1239
x4           1.21557    0.15086   8.058   0.0786 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.1601 on 1 degrees of freedom
Multiple R-squared:  0.9961,    Adjusted R-squared:  0.9805
F-statistic: 64.01 on 4 and 1 DF,  p-value: 0.09344


It fits the data almost perfectly e.g. try

plot(predict(overfit),y)

But it's all random noise.

If we try to apply this model to other data, we will get junk.

Every model has an error. The best model is that which minimizes the error associated with its predictions. This is why models are typically constructed using only a proportion of the data (in-sample), and then applied to the remaining 'out of sample' data set. An over-fitted model will typically have a greater prediction error in practice than a well formulated one. In addition, a model should be intellectually robust: there is no point constructing a model that works in one 'regime' if it does not work at all in the event of regime change. Such a model might appear to be very well formed until such time as the regime changes because essentially such a model has been constructed 'in-sample'. Another way of saying that is that the model's expected error must be well formulated too. There is also the matter of 'Occam's Razor', which is a philosophical idea that essentially the model should be the simplest possible, using the least number of variables required to describe the system being modelled. This serves as a useful guide, rather than a set-in-stone rule, but I believe that this is the idea behind using the 'adjusted R squared' rather than the R squared, to adjust for the natural improvement in fit associated with using more variables (e.g. you would have perfect fit, an R squared of 100% if you had a separate variable for every piece of data!). It is also an idea that should be applied to modern ML techniques: throwing e.g. thousands of variables at an ML algorithm is dangerous unless you have millions of pieces of data (and even then ... you might be better off transforming your data to reduce the number of variables first). One final point: every model requires belief. Even our laws of Physics are based on observation, and indeed they have required modification as we moved from Newtonian physics into the realms of the very small (Quantum mechanics) and the very large (General Relativity). We cannot say with absolute certainty that our current laws of Physics will hold in the future, or even in the past (e.g. around the time of the big bang). But appealing to our philosophical belief in Occam's razor results in us accepting these models and ideas because they are the simplest models yet devised that fit our observations and data.

In summary, there are no hard and fast rules. Imagine a complex (chaotic?) dynamical system, for example, the global economy. You might construct a well-formed model that works well for a short period of time. But 'regime change' is a very real issue: the economic system is highly complex and non-linear, and there are far more variables than you can measure, that might be of no consequence in the in-sample regime, but of huge significance in another 'regime'. But within your short, essentially in-sample, period, you might find that linear regression works quite well. Common sense should prevail: sometimes a very complex model is required, but it should be heavily caveated if the error associated with its predictions is unknown.

I'm sure that a proper statistician can give a much better answer than this, but since none of the above points seem to have been made yet, I thought that I would stick my neck out ...

All models are wrong, but some are less wrong than others.

Overfitting generally makes your model more wrong in dealing with real-world data.

If a doctor were to try to diagnose whether you have cancer, would you rather have them be wrong 50% of the time (very wrong) or 0.1% of the time (much less wrong)?

Or, let's say you give away something for free if your model predicts this will lead to the customer buying something later. Would you rather give away many things for free without this making a difference to whether customers buy things later (quite wrong) or have most customers come back to buy things later (less wrong)?

Clearly less wrong is better.