What's a real-world example of "overfitting"?

Question

I kind of understand what "overfitting" means, but I need help as to how to come up with a real-world example that applies to overfitting.

Answer 1

Here's a nice example of presidential election time series models from xkcd:

There have only been 56 presidential elections and 43 presidents. That is not a lot of data to learn from. When the predictor space expands to include things like having false teeth and the Scrabble point value of names, it's pretty easy for the model to go from fitting the generalizable features of the data (the signal) and to start matching the noise. When this happens, the fit on the historical data may improve, but the model will fail miserably when used to make inferences about future presidential elections.

Answer 2

My favorite was the Matlab example of US census population versus time:

• A linear model is pretty good
• A quadratic model is closer
• A quartic model predicts total annihilation starting next year

(At least I sincerely hope this is an example of overfitting)

http://www.mathworks.com/help/curvefit/examples/polynomial-curve-fitting.html#zmw57dd0e115

Answer 3

The study of Chen et al. (2013) fits two cubics to a supposed discontinuity in life expectancy as a function of latitude.

Chen Y., Ebenstein, A., Greenstone, M., and Li, H. 2013. Evidence on the impact of sustained exposure to air pollution on life expectancy from China's Huai River policy. Proceedings of the National Academy of Sciences 110: 12936–12941. abstract

Despite its publication in an outstanding journal, etc., its tacit endorsement by distinguished people, etc., I would still present this as a prima facie example of over-fitting.

A tell-tale sign is the implausibility of cubics. Fitting a cubic implicitly assumes there is some reason why life expectancy would vary as a third-degree polynomial of the latitude where you live. That seems rather implausible: it is not easy to imagine a plausible physical mechanism that would cause such an effect.

See also the following blog post for a more detailed analysis of this paper: Evidence on the impact of sustained use of polynomial regression on causal inference (a claim that coal heating is reducing lifespan by 5 years for half a billion people).

Answer 4

In a March 14, 2014 article in Science, David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani identified problems in Google Flu Trends that they attribute to overfitting.

Here is how they tell the story, including their explanation of the nature of the overfitting and why it caused the algorithm to fail:

In February 2013, ... Nature reported that GFT was predicting more than double the proportion of doctor visits for influenza-like illness (ILI) than the Centers for Disease Control and Prevention (CDC) ... . This happened despite the fact that GFT was built to predict CDC reports. ...

Essentially, the methodology was to find the best matches among 50 million search terms to fit 1152 data points. The odds of finding search terms that match the propensity of the flu but are structurally unrelated, and so do not predict the future, were quite high. GFT developers, in fact, report weeding out seasonal search terms unrelated to the flu but strongly correlated to the CDC data, such as those regarding high school basketball. This should have been a warning that the big data were overfitting the small number of cases—a standard concern in data analysis. This ad hoc method of throwing out peculiar search terms failed when GFT completely missed the nonseasonal 2009 influenza A–H1N1 pandemic.

[Emphasis added.]

Answer 5

I saw this image a few weeks ago and thought it was rather relevant to the question at hand.

Instead of linearly fitting the sequence, it was fitted with a quartic polynomial, which had perfect fit, but resulted in a clearly ridiculous answer.

Answer 6

To me the best example is Ptolemaic system in astronomy. Ptolemy assumed that Earth is at the center of the universe, and created a sophisticated system of nested circular orbits, which would explain movements of object on the sky pretty well. Astronomers had to keep adding circles to explain deviation, until one day it got so convoluted that folks started doubting it. That's when Copernicus came up with a more realistic model.

This is the best example of overfitting to me. You can't overfit data generating process (DGP) to the data. You can only overfit misspecified model. Almost all our models in social sciences are misspecified, so the key is to remember this, and keep them parsimonious. Not to try to catch every aspect of the data set, but try to capture the essential features through simplification.

Answer 7

Let's say you have 100 dots on a graph.

You could say: hmm, I want to predict the next one.

• with a line
• with a 2nd order polynomial
• with a 3rd order polynomial
• ...
• with a 100th order polynomial

Here you can see a simplified illustration for this example:

The higher the polynomial order, the better it will fit the existing dots.

However, the high order polynomials, despite looking like to be better models for the dots, are actually overfitting them. It models the noise rather than the true data distribution.

As a consequence, if you add a new dot to the graph with your perfectly fitting curve, it'll probably be further away from the curve than if you used a simpler low order polynomial.

Answer 8

The analysis that may have contributed to the Fukushima disaster is an example of overfitting. There is a well known relationship in Earth Science that describes the probability of earthquakes of a certain size, given the observed frequency of "lesser" earthquakes. This is known as the Gutenberg-Richter relationship, and it provides a straight-line log fit over many decades. Analysis of the earthquake risk in the vicinity of the reactor (this diagram from Nate Silver's excellent book "The Signal and the Noise") show a "kink" in the data. Ignoring the kink leads to an estimate of the annualized risk of a magnitude 9 earthquake as about 1 in 300 - definitely something to prepare for. However, overfitting a dual slope line (as was apparently done during the initial risk assessment for the reactors) reduces the risk prediction to about 1 in 13,000 years. One could not fault the engineers for not designing the reactors to withstand such an unlikely event - but one should definitely fault the statisticians who overfitted (and then extrapolated) the data...

Answer 9

"Agh! Pat is leaving the company. How are we ever going to find a replacement?"

Job Posting:

Wanted: Electrical Engineer. 42 year old androgynous person with degrees in Electrical Engineering, mathematics, and animal husbandry. Must be 68 inches tall with brown hair, a mole over the left eye, and prone to long winded diatribes against geese and misuse of the word 'counsel'.

In a mathematical sense, overfitting often refers to making a model with more parameters than are necessary, resulting in a better fit for a specific data set, but without capturing relevant details necessary to fit other data sets from the class of interest.

In the above example, the poster is unable to differentiate the relevant from irrelevant characteristics. The resulting qualifications are likely only met by the one person that they already know is right for the job (but no longer wants it).

Answer 10

This one is made-up, but I hope it will illustrate the case.

Example 1

First, let's make up some random data. Here you have $k=100$ variables, each drawn from a standard normal distribution, with $n=100$ cases:

set.seed(123)
k <- 100
data <- replicate(k, rnorm(100))
colnames(data) <- make.names(1:k)
data <- as.data.frame(data)

Now, let's fit a linear regression to it:

fit <- lm(X1 ~ ., data=data)

And here is a summary for first ten predictors:

> summary(fit)

Call:
lm(formula = X1 ~ ., data = data)

Residuals:
ALL 100 residuals are 0: no residual degrees of freedom!

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept) -1.502e-01         NA      NA       NA
X2           3.153e-02         NA      NA       NA
X3          -6.200e-01         NA      NA       NA
X4           7.087e-01         NA      NA       NA
X5           4.392e-01         NA      NA       NA
X6           2.979e-01         NA      NA       NA
X7          -9.092e-02         NA      NA       NA
X8          -5.783e-01         NA      NA       NA
X9           5.965e-01         NA      NA       NA
X10         -8.289e-01         NA      NA       NA
...
Residual standard error: NaN on 0 degrees of freedom
Multiple R-squared:      1, Adjusted R-squared:    NaN 
F-statistic:   NaN on 99 and 0 DF,  p-value: NA

results look pretty weird, but let's plot it.

That is great, fitted values perfectly fit the $X_1$ values. Error variance is literally zero. But, let it not convince us, let's check what is the sum of absolute differences between $X_1$ and fitted values:

> sum(abs(data$X1-fitted(fit)))
[1] 0

It is zero, so the plots were not lying to us: the model fits perfectly. And how precise is it in classification?

> sum(data$X1==fitted(fit))
[1] 100

We get 100 out of 100 fitted values that are identical to $X_1$. And we got this with totally random numbers fitted to other totally random numbers.

Example 2

One more example. Lets make up some more data:

data2 <- cbind(1:10, diag(10))
colnames(data2) <- make.names(1:11)
data2 <- as.data.frame(data2)

so it looks like this:

   X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11
1   1  1  0  0  0  0  0  0  0   0   0
2   2  0  1  0  0  0  0  0  0   0   0
3   3  0  0  1  0  0  0  0  0   0   0
4   4  0  0  0  1  0  0  0  0   0   0
5   5  0  0  0  0  1  0  0  0   0   0
6   6  0  0  0  0  0  1  0  0   0   0
7   7  0  0  0  0  0  0  1  0   0   0
8   8  0  0  0  0  0  0  0  1   0   0
9   9  0  0  0  0  0  0  0  0   1   0
10 10  0  0  0  0  0  0  0  0   0   1

and now lets fit a linear regression to this:

fit2 <- lm(X1~., data2)

so we get following estimates:

> summary(fit2)

Call:
lm(formula = X1 ~ ., data = data2)

Residuals:
ALL 10 residuals are 0: no residual degrees of freedom!

Coefficients: (1 not defined because of singularities)
            Estimate Std. Error t value Pr(>|t|)
(Intercept)       10         NA      NA       NA
X2                -9         NA      NA       NA
X3                -8         NA      NA       NA
X4                -7         NA      NA       NA
X5                -6         NA      NA       NA
X6                -5         NA      NA       NA
X7                -4         NA      NA       NA
X8                -3         NA      NA       NA
X9                -2         NA      NA       NA
X10               -1         NA      NA       NA
X11               NA         NA      NA       NA

Residual standard error: NaN on 0 degrees of freedom
Multiple R-squared:      1, Adjusted R-squared:    NaN 
F-statistic:   NaN on 9 and 0 DF,  p-value: NA

as you can see, we have $R^2 = 1$, i.e. "100% variance explained". Linear regression didn't even need to use 10th predictor. From this regression we see, that $X_1$ can be predicted using function:

$$X_1 = 10 + X_2 \times -9 + X_3 \times -8 + X_4 \times -7 + X_5 \times -6 + X_6 \times -5 + X_7 \times -4 + X_8 \times -3 + X_9 \times -2$$

so $X_1 = 1$ is:

$$10 + 1 \times -9 + 0 \times -8 + 0 \times -7 + 0 \times -6 + 0 \times -5 + 0 \times -4 + 0 \times -3 + 0 \times -2$$

It is pretty self-explanatory. You can think of Example 1 as similar to Example 2 but with some "noise" added. If you have big enough data and use it for "predicting" something then sometimes a single "feature" may convince you that you have a "pattern" that describes your dependent variable well, while it could be just a coincidence. In Example 2 nothing is really predicted, but exactly the same has happened in Example 1 just the values of the variables were different.

Real life examples

The real life example for this is predicting terrorist attacks on 11 September 2001 by watching "patterns" in numbers randomly drawn by computer pseudorandom number generators by Global Consciousness Project or "secret messages" in "Moby Dick" that reveal facts about assassinations of famous people (inspired by similar findings in Bible).

Conclusion

If you look hard enough, you'll find "patterns" for anything. However, those patterns won't let you learn anything about the universe and won't help you reach any general conclusions. They will fit perfectly to your data, but would be useless since they won't fit anything else then the data itself. They won't let you make any reasonable out-of-sample predictions, because what they would do, is they would rather imitate than describe the data.

Answer 11

A common problem that results in overfitting in real life is that in addition to terms for a correctly specified model, we may have have added something extraneous: irrelevant powers (or other transformations) of the correct terms, irrelevant variables, or irrelevant interactions.

This happens in multiple regression if you add a variable that should not appear in the correctly specified model but do not want to drop it because you are afraid of inducing omitted variable bias. Of course, you have no way of knowing you have wrongly included it, since you can't see the whole population, only your sample, so can't know for sure what the correct specification is. (As @Scortchi points out in the comments, there may be no such thing as a the "correct" model specification - in that sense, the aim of modelling is finding a "good enough" specification; to avoid overfitting involves avoiding a model complexity greater than can be sustained from the available data.) If you want a real-world example of overfitting, this happens every time you throw all the potential predictors into a regression model, should any of them in fact have no relationship with the response once the effects of others are partialled out.

With this type of overfitting, the good news is that inclusion of these irrelevant terms does not introduce bias of your estimators, and in very large samples the coefficients of the irrelevant terms should be close to zero. But there is also bad news: because the limited information from your sample is now being used to estimate more parameters, it can only do so with less precision - so the standard errors on the genuinely relevant terms increase. That also means they're likely to be further from the true values than estimates from a correctly specified regression, which in turn means that if given new values of your explanatory variables, the predictions from the overfitted model will tend to be less accurate than for the correctly specified model.

Here is a plot of log GDP against log population for 50 US states in 2010. A random sample of 10 states was selected (highlighted in red) and for that sample we fit a simple linear model and a polynomial of degree 5. For the sample points, the polynomial has extra degrees of freedom that let it "wriggle" closer to the observed data than the straight line can. But the 50 states as a whole obey a nearly linear relationship, so the predictive performance of the polynomial model on the 40 out-of-sample points is very poor compared to the less complex model, particularly when extrapolating. The polynomial was effectively fitting some of the random structure (noise) of the sample, which did not generalise to the wider population. It was particularly poor at extrapolating beyond the observed range of the sample. (Code plus data for this plot is at the bottom of this revision of this answer.)

Similar issues affect regression against multiple predictors. To look at some actual data, it's easier with simulation rather than real-world samples since this way you control the data-generating process (effectively, you get to see the "population" and the true relationship). In this R code, the true model is $y_i = 2x_{1,i} + 5 + \epsilon_i$ but data is also provided on irrelevant variables $x_2$ and $x_3$. I have designed the simulation so that the the predictor variables are correlated, which would be a common occurrence in real-life data. We fit models which are correctly specified and overfitted (includes the irrelevant predictors and their interactions) on one portion of the generated data, then compare predictive performance on a holdout set. The multicollinearity of the predictors makes life even harder for the overfitted model, since it becomes harder for it to pick apart the effects of $x_1$, $x_2$ and $x_3$, but note that this does not bias any of the coefficient estimators.

require(MASS) #for multivariate normal simulation    
nsample <- 25   #sample to regress 
nholdout <- 1e6  #to check model predictions
Sigma <- matrix(c(1, 0.5, 0.4, 0.5, 1, 0.3, 0.4, 0.3, 1), nrow=3)
df <- as.data.frame(mvrnorm(n=(nsample+nholdout), mu=c(5,5,5), Sigma=Sigma))
colnames(df) <- c("x1", "x2", "x3")
df$y <- 5 + 2 * df$x1 + rnorm(n=nrow(df)) #y = 5 + *x1 + e

holdout.df <- df[1:nholdout,]
regress.df <- df[(nholdout+1):(nholdout+nsample),]

overfit.lm <- lm(y ~ x1*x2*x3, regress.df)
correctspec.lm <- lm(y ~ x1, regress.df)
summary(overfit.lm)
summary(correctspec.lm)

holdout.df$overfitPred <- predict.lm(overfit.lm, newdata=holdout.df)
holdout.df$correctSpecPred <- predict.lm(correctspec.lm, newdata=holdout.df)
with(holdout.df, sum((y - overfitPred)^2)) #SSE
with(holdout.df, sum((y - correctSpecPred)^2))

require(ggplot2)
errors.df <- data.frame(
    Model = rep(c("Overfitted", "Correctly specified"), each=nholdout),
    Error = with(holdout.df, c(y - overfitPred, y - correctSpecPred)))
ggplot(errors.df, aes(x=Error, color=Model)) + geom_density(size=1) +
    theme(legend.position="bottom")

Here are my results from one run, but it's best to run the simulation several times to see the effect of different generated samples.

>     summary(overfit.lm)

Call:
lm(formula = y ~ x1 * x2 * x3, data = regress.df)

Residuals:
     Min       1Q   Median       3Q      Max 
-2.22294 -0.63142 -0.09491  0.51983  2.24193 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) 18.85992   65.00775   0.290    0.775
x1          -2.40912   11.90433  -0.202    0.842
x2          -2.13777   12.48892  -0.171    0.866
x3          -1.13941   12.94670  -0.088    0.931
x1:x2        0.78280    2.25867   0.347    0.733
x1:x3        0.53616    2.30834   0.232    0.819
x2:x3        0.08019    2.49028   0.032    0.975
x1:x2:x3    -0.08584    0.43891  -0.196    0.847

Residual standard error: 1.101 on 17 degrees of freedom
Multiple R-squared: 0.8297,     Adjusted R-squared: 0.7596 
F-statistic: 11.84 on 7 and 17 DF,  p-value: 1.942e-05

These coefficient estimates for the overfitted model are terrible - should be about 5 for the intercept, 2 for $x_1$ and 0 for the rest. But the standard errors are also large. The correct values for those parameters do lie well within the 95% confidence intervals in each case. The $R^2$ is 0.8297 which suggests a reasonable fit.

>     summary(correctspec.lm)

Call:
lm(formula = y ~ x1, data = regress.df)

Residuals:
    Min      1Q  Median      3Q     Max 
-2.4951 -0.4112 -0.2000  0.7876  2.1706 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   4.7844     1.1272   4.244 0.000306 ***
x1            1.9974     0.2108   9.476 2.09e-09 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 1.036 on 23 degrees of freedom
Multiple R-squared: 0.7961,     Adjusted R-squared: 0.7872 
F-statistic:  89.8 on 1 and 23 DF,  p-value: 2.089e-09

The coefficient estimates are much better for the correctly specified model. But note that the $R^2$ is lower, at 0.7961, as the less complex model has less flexibility in fitting the observed responses. $R^2$ is more dangerous than useful in this case!

>     with(holdout.df, sum((y - overfitPred)^2)) #SSE
[1] 1271557
>     with(holdout.df, sum((y - correctSpecPred)^2))
[1] 1052217

The higher $R^2$ on the sample we regressed on showed how the overfitted model produced predictions, $\hat{y}$, that were closer to the observed $y$ than the correctly specified model could. But that's because it was overfitting to that data (and had more degrees of freedom to do so than the correctly specified model did, so could produce a "better" fit). Look at the Sum of Squared Errors for the predictions on the holdout set, which we didn't use to estimate the regression coefficients from, and we can see how much worse the overfitted model has performed. In reality the correctly specified model is the one which makes the best predictions. We shouldn't base our assessment of the predictive performance on the results from the set of data we used to estimate the models. Here's a density plot of the errors, with the correct model specification producing more errors close to 0:

The simulation clearly represents many relevant real-life situations (just imagine any real-life response which depends on a single predictor, and imagine including extraneous "predictors" into the model) but has the benefit that you can play with the data-generating process, the sample sizes, the nature of the overfitted model and so on. This is the best way you can examine the effects of overfitting since for observed data you don't generally have access to the DGP, and it's still "real" data in the sense that you can examine and use it. Here are some worthwhile ideas that you should experiment with:

• Run the simulation several times and see how the results differ. You will find more variability using small sample sizes than large ones.
• Try changing the sample sizes. If increased to, say, n <- 1e6, then the overfitted model eventually estimates reasonable coefficients (about 5 for intercept, about 2 for $x_1$, about 0 for everything else) and its predictive performance as measured by SSE doesn't trail the correctly specified model so badly. Conversely, try fitting on a very small sample (bear in mind you need to leave enough degrees of freedom to estimate all the coefficients) and you will see that the overfitted model has appalling performance both for estimating coefficients and predicting for new data.
• Try reducing the correlation between the predictor variables by playing with the off-diagonal elements of the variance-covariance matrix Sigma. Just remember to keep it positive semi-definite (which includes being symmetric). You should find if you reduce the multicollinearity, the overfitted model doesn't perform quite so badly. But bear in mind that correlated predictors do occur in real life.
• Try experimenting with the specification of the overfitted model. What if you include polynomial terms?
• What if you simulate data for a different region of predictors, rather than having their mean all around 5? If the correct data generating process for $y$ is still df$y <- 5 + 2*df$x1 + rnorm(n=nrow(df)), see how well the models fitted to the original data can predict that $y$. Depending on how you generate the $x_i$ values, you may find that extrapolation with the overfitted model produces predictions far worse than the correctly specified model.
• What if you change the data generating process so that $y$ now depends, weakly, on $x_2$, $x3$ and perhaps the interactions as well? This may be a more realistic scenario that depending on $x_1$ alone. If you use e.g. df$y <- 5 + 2 * df$x1 + 0.1*df$x2 + 0.1*df$x3 + rnorm(n=nrow(df)) then $x_2$ and $x_3$ are "almost irrelevant", but not quite. (Note that I drew all the $x$ variables from the same range, so it does make sense to compare their coefficients like that.) Then the simple model involving only $x_1$ suffers omitted variable bias, though since $x_2$ and $x_3$ are not particularly important, this is not too severe. On a small sample, e.g. nsample <- 25, the full model is still overfitted, despite being a better representation of the underlying population, and on repeated simulations its predictive performance on the holdout set is still consistently worse. With such limited data, it's more important to get a good estimate for the coefficient of $x_1$ than to expend information on the luxury of estimating the less important coefficients. With the effects of $x_2$ and $x_3$ being so hard to discern in a small sample, the full model is effectively using the flexibility from its extra degrees of freedom to "fit the noise" and this generalises poorly. But with nsample <- 1e6, it can estimate the weaker effects pretty well, and simulations show the complex model has predictive power that outperforms the simple one. This shows how "overfitting" is an issue of both model complexity and the available data.

Answer 12

A form of overfitting is fairly common in sports, namely to identify patterns to explain past results by factors that have no or at best vague power to predict future results. A common feature of these "patterns" is that they are often based on very few cases so that pure chance is probably the most plausible explanation for the pattern.

Examples include things like (the "quotes" are made up by me, but often look similar)

Team A has won all X games since the coach has starting wearing his magical red jacket.

Similar:

We shall not be shaving ourselves during the playoffs, because that has helped us win the past X games.

Less superstitious, but a form of overfitting as well:

Borussia Dortmund has never lost a Champions League home game to a Spanish opponent when they have lost the previous Bundesliga away game by more than two goals, having scored at least once themselves.

Similar:

Roger Federer has won all his Davis Cup appearances to European opponents when he had at least reached the semi-finals in that year's Australian Open.

The first two are fairly obvious nonsense (at least to me). The last two examples may perfectly well hold true in sample (i.e., in the past), but I would be most happy to bet against an opponent who would let this "information" substantially affect his odds for Dortmund beating Madrid if they lost 4:1 at Schalke on the previous Saturday or Federer beating Djokovic, even if he won the Australian Open that year.

Answer 13

When I was trying to understand this myself, I started thinking in terms of analogies with describing real objects, so I guess it's as "real world" as you can get, if you want to understand the general idea:

Say you want to describe to someone the concept of a chair, so that they get a conceptual model that allows them to predict if a new object they find is a chair. You go to Ikea and get a sample of chairs, and start describing them by using two variables: it's an object with 4 legs where you can sit. Well, that may also describe a stool or a bed or a lot of other things. Your model is underfitting, just as if you were to try and model a complex distribution with too few variables - a lot of non-chair things will be identified as chairs. So, let's increase the number of variables, add that the object has to have a back, for example. Now you have a pretty acceptable model that describes your set of chairs, but is general enough to allow a new object to be identified as one. Your model describes the data, and is able to make predictions. However, say you happen to have got a set where all chairs are black or white, and made of wood. You decide to include those variables in your model, and suddenly it won't identify a plastic yellow chair as a chair. So, you've overfitted your model, you have included features of your dataset as if they were features of chairs in general, (if you prefer, you have identified "noise" as "signal", by interpreting random variation from your sample as a feature of the whole "real world chairs"). So, you either increase your sample and hope to include some new material and colors, or decrease the number of variables in your models.

This may be a simplistic analogy and breakdown under further scrutiny, but I think it works as a general conceptualization... Let me know if some part needs clarification.

Answer 14

In predictive modeling, the idea is to use the data at hand to discover the trends that exist and that can be generalized to future data. By including variables in your model that have some minor, non-significant effect you are abandoning this idea. What you are doing is considering the specific trends in your specific sample that are only there because of random noise instead of a true, underlying trend. In other words, a model with too many variables fits the noise rather than discovering the signal.

Here's an exaggerated illustration of what I'm talking about. Here the dots are the observed data and the line is our model. Look at that a perfect fit - what a great model! But did we really discover the trend or are we just fitting to the noise? Likely the latter.

Answer 15

Here is a "real world" example not in the sense that somebody happened to come across it in research, but in the sense that it uses everyday concepts without many statistic-specific terms. Maybe this way of saying it will be more helpful for some people whose training is in other fields.

Imagine that you have a database with data about patients with a rare disease. You are a medical graduate student and want to see if you can recognize risk factors for this disease. There have been 8 cases of the disease in this hospital, and you have recorded 100 random pieces of information about them: age, race, birth order, have they had measles as a child, whatever. You also have recorded the data for 8 patients without this disease.

You decide to use the following heuristic for risk factors: if a factor takes a given value in more than one of your diseased patients, but in 0 of your controls, you will consider it a risk factor. (In real life, you'd use a better method, but I want to keep it simple). You find out that 6 of your patients are vegetarians (but none of the controls is vegetarian), 3 have Swedish ancestors, and two of them have a stuttering speech impairment. Out of the other 97 factors, there is nothing which occurs in more than one patient, but is not present among the controls.

Years later, somebody else takes interest in this orphan disease and replicates your research. Because he works at a larger hospital, which has a data-sharing cooperation with other hospitals, he can use data about 106 cases, as opposed to your 8 cases. And he finds out that the prevalence of stutterers is the same in the patient group and the control group; stuttering is not a risk factor.

What happened here is that your small group had 25% stutterers by random chance. Your heuristic had no way of knowing if this is medically relevant or not. You gave it criteria to decide when you consider a pattern in the data "interesting" enough to be included in the model, and according to these criteria, the stuttering was interesting enough.

Your model has been overfitted, because it mistakenly included a parameter which is not really relevant in the real world. It fits your sample - the 8 patients + 8 controls - very well, but it does not fit the real world data. When a model describes your sample better than it describes reality, it's called overfitted.

Had you chosen a threshold of 3 out of 8 patients having a feature, it wouldn't have happened - but you'd had a higher chance to miss something actually interesting. Especially in medicine, where many diseases only happen in a small fraction of people exhibiting in risk factor, that's a hard trade-off to make. And there are methods to avoid it (basically, compare to a second sample and see if the explaining power stays the same or falls), but this is a topic for another question.

Answer 16

Here's a real-life example of overfitting that I helped perpetrate and then tried (unsuccessfully) to avert:

I had several thousand independent, bivariate time series, each with no more than 50 data points, and the modeling project involved fitting a vector autoregression (VAR) to each one. No attempt was made to regularize across observations, estimate variance components, or anything like that. The time points were measured over the course of a single year, so the data were subject to all kinds of seasonal and cyclical effects that only appeared once in each time series.

One subset of the data exhibited an implausibly high rate of Granger causality compared to the rest of the data. Spot checks revealed that positive spikes were occurring one or two lags apart in this subset, but it was clear from the context that both spikes were caused directly by an external source and that one spike was not causing the other. Out-of-sample forecasts using this models would probably be quite wrong, because the models were overfitted: rather than "smoothing out" the spikes by averaging them into the rest of the data, there were few enough observations that the spikes were actually driving the estimates.

Overall, I don't think the project went badly but I don't think it produced results that were anywhere near as useful as they could have been. Part of the reason for this is that the many-independent-VARs procedure, even with just one or two lags, was having a hard time distinguishing between data and noise, and so was fitting to the latter at the expense of providing insight about the former.

Answer 17

My favourite is the “3964 formula” discovered before the World Cup soccer competition in 1998:

Brazil won the championships in 1970 and 1994. Sum up these 2 numbers and you will get 3964; Germany won in 1974 and 1990, adding up again to 3964; the same thing with Argentina winning in 1978 and 1986 (1978+1986 = 3964).

This is a very surprising fact, but everyone can see that it is not advisable to base any future prediction on that rule. And indeed, the rule gives that the winner of the World Cup in 1998 should have been England since 1966 + 1998 = 3964 and England won in 1966. This didn’t happen and the winner was France.

Answer 18

Studying for an exam by memorising the answers to last year's exam.

Answer 19

Many intelligent people in this thread --- many much more versed in statistics than I am. But I still don't see an easy-to-understand to the lay-person example. The Presidential example doesn't quite hit the bill in terms of typical overfitting, because while it is technically overfitting in each one of its wild claims, usually an overfitting model overfits -ALL- the given noise, not just one element of it.

I really like the chart in the bias-variance tradeoff explanation in wikipedia: http://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff

(The lowermost chart is the example of overfitting).

I'm hard pressed to think of a real world example that doesn't sound like complete mumbo-jumbo. The idea is that data is part caused by measurable, understandable variables --- part random noise. Attempting to model this noise as a pattern gives you inaccuracy.

A classic example is modeling based SOLELY on R^2 in MS Excel (you are attempting to fit an equation/ model literally as close as possible to the data using polynomials, no matter how nonsensical).

Say you're trying to model ice cream sales as a function of temperature. You have "real world" data. You plot the data and try to maximize R^2. You'll find using real-world data, the closest fit equation is not linear or quadratic (which would make logical sense). Like almost all equations, the more nonsensical polynomial terms you add (x^6 -2x^5 +3x^4+30x^3-43.2x^2-29x) -- the closer it fits the data. So how does that sensibly relate temperature to ice cream sales? How would you explain that ridiculous polynomial? Truth is, it's not the true model. You've overfit the data.

You are taking unaccounted for noise -- which may have been due to sales promotions or some other variable or "noise" like a butterfly flapping its wings in the cosmos (something never predictable)--- and attempted to model that based on temperature. Now usually if your noise/ error does not average to zero or is auto-correlated, etc, it means there are more variables out there --- and then eventually you get to generally randomly distributed noise, but still, that's the best I can explain it.

Answer 20

Most optimization methods have some fudge factors aka hyperparameters. A real example:

For all systems under study, the following parameters yielded a fast and robust behavior: $N_{min} = 5,\ \ f_{inc} = 1.1,\ \ f_{dec} = 0.5,\ \ \alpha_{start} = 0.1, \ \ f_{\alpha} = 0.99.$

Is this over fitting, or just fitting to a particular set of problems ?

Answer 21

A bit intuitive, but maybe it'll help. Let's say you want to learn some new language. How do you learn? instead of learning the rules in a course, you use examples. Specifically, TV shows. So you like crime shows, and you watch few series of some cop show. Then, you take another crime show and watch some series form that one. By the third show you see - you know almost everything, no problem. You don't need the English subtitles.

But then you try your newly learned language on the street on your next visit, and you realize you can't talk about anything other than saying "officer! that man took my bag and shot that lady!". While your 'training error' was zero, your 'test error' is high, due to 'overfitting' the language, studying only a limited subset of words and assuming its enough.