96

Here's a nice example of presidential election time series models from xkcd: There have only been 56 presidential elections and 43 presidents. That is not a lot of data to learn from. When the predictor space expands to include things like having false teeth and the Scrabble point value of names, it's pretty easy for the model to go from fitting the ...


82

My favorite was the Matlab example of US census population versus time: A linear model is pretty good A quadratic model is closer A quartic model predicts total annihilation starting next year (At least I sincerely hope this is an example of overfitting) http://www.mathworks.com/help/curvefit/examples/polynomial-curve-fitting.html#zmw57dd0e115


56

The quote by Box is along the lines of "All models are wrong, but some are useful." If we have bad overfitting, our model will not be useful in making predictions on new data.


49

The study of Chen et al. (2013) fits two cubics to a supposed discontinuity in life expectancy as a function of latitude. Chen Y., Ebenstein, A., Greenstone, M., and Li, H. 2013. Evidence on the impact of sustained exposure to air pollution on life expectancy from China's Huai River policy. Proceedings of the National Academy of Sciences 110: 12936–12941. ...


44

Summary: PCA can be performed before LDA to regularize the problem and avoid over-fitting. Recall that LDA projections are computed via eigendecomposition of $\boldsymbol \Sigma_W^{-1} \boldsymbol \Sigma_B$, where $\boldsymbol \Sigma_W$ and $\boldsymbol \Sigma_B$ are within- and between-class covariance matrices. If there are less than $N$ data points (...


38

In a March 14, 2014 article in Science, David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani identified problems in Google Flu Trends that they attribute to overfitting. Here is how they tell the story, including their explanation of the nature of the overfitting and why it caused the algorithm to fail: In February 2013, ... Nature reported ...


36

Yes, you can overfit logistic regression models. But first, I'd like to address the point about the AUC (Area Under the Receiver Operating Characteristic Curve): There are no universal rules of thumb with the AUC, ever ever ever. What the AUC is is the probability that a randomly sampled positive (or case) will have a higher marker value than a negative (...


35

How are you getting that 99% AUC on your training data? Be aware that there's a difference between predict(model) and predict(model, newdata=train) when getting predictions for the training dataset. The first option gets the out-of-bag predictions from the random forest. This is generally what you want, when comparing predicted values to actuals on the ...


35

Is overfitting so bad that you should not pick a model that does overfit, even though its test error is smaller? No. But you should have a justification for choosing it. This behavior is not restricted to XGBoost. It is a common thread among all machine learning techniques; finding the right tradeoff between underfitting and overfitting. The formal ...


33

I think the argument is correct. If 70% is acceptable in the particular application, then the model is useful even though it is overfitted (more generally, regardless of whether it is overfitted or not). While balancing overfitting against underfitting concerns optimality (looking for an optimal solution), having satisfactory performance is about ...


32

To avoid over-fitting in random forest, the main thing you need to do is optimize a tuning parameter that governs the number of features that are randomly chosen to grow each tree from the bootstrapped data. Typically, you do this via $k$-fold cross-validation, where $k \in \{5, 10\}$, and choose the tuning parameter that minimizes test sample prediction ...


32

I saw this image a few weeks ago and thought it was rather relevant to the question at hand. Instead of linearly fitting the sequence, it was fitted with a quartic polynomial, which had perfect fit, but resulted in a clearly ridiculous answer.


31

Why do we worry about overfitting even if “all models are wrong”? Your question appears to be a variation of the Nirvana fallacy, implicitly suggesting that if there is no perfect model, then every model is equally satisfactory (and therefore flaws in models are irrelevant). Observe that you could just as easily ask this same question about any flaw in a ...


30

I'll try to answer in the simplest way. Each of those problems has its own main origin: Overfitting: Data is noisy, meaning that there are some deviations from reality (because of measurement errors, influentially random factors, unobserved variables and rubbish correlations) that makes it harder for us to see their true relationship with our explaining ...


26

In my past project with Credit Card Fraud detection, we intentionally want to over fit the data / hard coded to remember fraud cases. (Note, overfitting one class is not exactly the general overfitting problem OP talked about.) Such system has relatively low false positives and satisfy our needs. So, I would say, overfitted model can be useful for some ...


26

A natural regularization happens because of the presence of many small components in the theoretical PCA of $x$. These small components are implicitly used to fit the noise using small coefficients. When using minimum norm OLS, you fit the noise with many small independent components and this has a regularizing effect equivalent to Ridge regularization. This ...


25

No, it is not true. Bayesian methods will certainly overfit the data. There are a couple of things that make Bayesian methods more robust against overfitting and you can make them more fragile as well. The combinatoric nature of Bayesian hypotheses, rather than binary hypotheses allows for multiple comparisons when someone lacks the "true" model for null ...


23

Suppose you are trying to minimize the objective function via number of iterations. And current value is $100.0$. In given data set, there are no "irreducible errors" and you can minimize the loss to $0.0$ for your training data. Now you have two ways to do it. The first way is "large learning rate" and few iterations. Suppose you can reduce loss by $10.0$ ...


23

Yes there is a (slightly more) rigorous definition: Given a model with a set of parameters, the model can be said to be overfitting the data if after a certain number of training steps, the training error continues to decrease while the out of sample (test) error starts increasing. In this example out of sample (test/validation) error first decreases in ...


22

To me the best example is Ptolemaic system in astronomy. Ptolemy assumed that Earth is at the center of the universe, and created a sophisticated system of nested circular orbits, which would explain movements of object on the sky pretty well. Astronomers had to keep adding circles to explain deviation, until one day it got so convoluted that folks started ...


22

Let's say you have 100 dots on a graph. You could say: hmm, I want to predict the next one. with a line with a 2nd order polynomial with a 3rd order polynomial ... with a 100th order polynomial Here you can see a simplified illustration for this example: The higher the polynomial order, the better it will fit the existing dots. However, the high order ...


21

training error (as in predict(model, data=train)) is typically useless. Unless you do (non-standard) pruning of the trees, it cannot be much above 0 by design of the algorithm. Random forest uses bootstrap aggregation of decision trees, which are known to be overfit badly. This is like training error for a 1-nearest-neighbour classifier. However, the ...


20

Not at all. However, cross validation helps you to assess by how much your method overfits. For instance, if your training data R-squared of a regression is 0.50 and the crossvalidated R-squared is 0.48, you hardly have any overfitting and you feel good. On the other hand, if the crossvalidated R-squared is only 0.3 here, then a considerable part of your ...


20

The full quote is "All models are wrong, but some are useful". We care about overfitting, because we still want our models to be useful. If you are familiar with the Bias-variance tradeoff, the "all models are wrong" statement is roughly equivalent to saying "all models have non-zero bias". Overfitting is the issue that while we can increase the number of ...


19

Optimisation is the root of all evil in statistics. Any time you make choices about your model$^1$ by optimising some suitable criterion evaluated on a finite sample of data you run the risk of over-fitting the criterion, i.e. reducing the statistic beyond the point where improvements in generalisation performance are obtained and the reduction is instead ...


19

The analysis that may have contributed to the Fukushima disaster is an example of overfitting. There is a well known relationship in Earth Science that describes the probability of earthquakes of a certain size, given the observed frequency of "lesser" earthquakes. This is known as the Gutenberg-Richter relationship, and it provides a straight-line log fit ...


19

Accuracy of a set is evaluated by just cross-checking the highest softmax output and the correct labeled class.It is not depended on how high is the softmax output. To make it clearer, here are some numbers. Suppose there are 3 classes- dog, cat and horse. For our case, the correct class is horse . Now, the output of the softmax is [0.9, 0.1]. For this ...


18

When you obtain a singular fit, this is often indicating that the model is overfitted – that is, the random effects structure is too complex to be supported by the data, which naturally leads to the advice to remove the most complex part of the random effects structure (usually random slopes). The benefit of this approach is that it leads to a more ...


17

People do that all the time with large networks. For example, the famous AlexNet network has about 60 million parameters, while the ImageNet ILSVRC it was originally trained on has only 1.2 million images. The reason you don't fit a 5-parameter polynomial to 4 data points is that it can always find a function that exactly fits your data points, but does ...


16

"Agh! Pat is leaving the company. How are we ever going to find a replacement?" Job Posting: Wanted: Electrical Engineer. 42 year old androgynous person with degrees in Electrical Engineering, mathematics, and animal husbandry. Must be 68 inches tall with brown hair, a mole over the left eye, and prone to long winded diatribes against geese and misuse of ...


Only top voted, non community-wiki answers of a minimum length are eligible