1
$\begingroup$

What exactly is overfitting while building models ?

$\endgroup$

3 Answers 3

1
$\begingroup$

I like @RRMT's answer, but I would like to emphasize this problem: adding many variables could look like it has a significant impact on the prediction power of the model, because it does... only for the data on hand.

Overfitting refers to creating a model trying to explain everything which is happening in the data, using way too many parameters/features compared to what is truly relevant for the model. Usually, the predictions of the model in the training data get better and better as we add (useless) variables, but the out-of-sample predictions could get worse and worse, or at least stagnate, while our expectations were higher because of our accuracy in-sample.

For example, imagine that you have the following relation:

$$Y_i=1+X_i+2X_i^2+\epsilon_i$$ with $X_i =-11+i, i=1,...,21$, and $\epsilon_i\sim N(0,\sigma^2=20^2)$

The data I simulated looks like this:

enter image description here

The green line represents $E(Y_i\vert X_i)$, i.e. the true model without the random errors.

If we try to fit the data using only a linear regression with $X$ as a predictor, then we create a simple line. The model underfits because the regressors we use are not sufficient to understand the underlying logic of the data.

enter image description here

If we add $X^2$ as a regressor, then our fit is very good, because the regressors we used explain very well the actual underlying logic of the data.

enter image description here

However, if you could add many regressors $X^k,k\in\{3,4,...,20\}$, you would end up creating a model which makes perfect predictions because you would have as many regressors as observed data points. The model would then overfit because it fits the data on hand perfectly, but way too much to really understands the logic of the data. The prediction curve (I failed to draw it because my regression becomes unstable due to the high numbers in the regressors) would go through all the observed points $\{x_i,y_i\}$ and it would make big swings between points, making out-of-sample predictions very bad.

I suggest you look at this article for a nice visual example.

EDIT:

I was able to add regressors up to $X^7$, here is what it looks like: enter image description here

You can see that there is a first sign of overfitting at the last data point: the regression overfits and make the predicition very close to the actual point whereas it should not be that close: we observe a deviation between the green line and the red line, due to overfitting, which corresponds to a worsening of the out-of-sample prediction accuracy in that area.

EDIT:

@RRMT proposed solutions which can be applied in the training data. In addition to these solutions, you can divide your dataset into a training set and a testing set in order to apply some backtesting procedures (such as cross-validation) in order to try to identify is some overfitting is occuring.

EDIT 2:

I changed my data (but kept the general concepts: quadratic true model vs high order polynonial (15)). Here is the result:

enter image description here

You can see that using many predictors allowed the model to create a prediction curve which comes extremely close to all the observations. But this accuracy is only artificial, it creates deviations from the true model. This is an example of overfitting.

In addition, I suggest you have a look the interesting comments in the comments below my answer.

$\endgroup$
5
  • $\begingroup$ Thanks for your answer. So, remedies are used to reduce number of parameters. So that only meaningful parameter's are used as regressor to avoid overfitting. To name these remedies are as mentioned by @RRMT $\endgroup$
    – SR1
    Jul 27, 2022 at 2:12
  • $\begingroup$ Yes, and I added a couple of sentence to propose other solutions. $\endgroup$
    – FP0
    Jul 27, 2022 at 8:33
  • $\begingroup$ This conveys the essence but "too much" (meaning, too many predictors) is the main idea of overfitting while "way too much" conveys extreme cases. As often, a definition can be a fairly useless paraphrase: a definition of fitting an excessively complicated model raises just as many questions, chiefly how to recognise that you are doing it Toy examples like that here make the point but don't really help learners much. That is, if the real model is quadratic plus noise, then clearly it's not a good idea to fit a more complicated polynomial. But this may not be the issue in practice. $\endgroup$
    – Nick Cox
    Jul 27, 2022 at 9:59
  • $\begingroup$ In some fields it is standard and even encouraged to fit many predictors that may have small effects even if they don't qualify as significant for various reasons e.g. they are part of a team or to show that you have taken account of a predictor's influence even if its effect is minor. There are many variants on this: e.g. including the complete set of indicators (dummy variables) for a categorical variable is often encouraged, as is always including main effects as well as interactions even if the main effect appears small. Simplicity can be a matter of statistical style too. $\endgroup$
    – Nick Cox
    Jul 27, 2022 at 10:01
  • $\begingroup$ @NickCox Thanks for your feedback. I agree with your points. I agree that my answer is not perfect and use a very specific example, but I think that extreme examples can help to grasp what could go (really) wrong in a particular case in order to understand the general concept/definition of overfitting. Ideally I would have liked to draw the predictions based on a polynomial of degree 20 to represent the real mess due to overfitting, but failed to do so. $\endgroup$
    – FP0
    Jul 27, 2022 at 10:43
0
$\begingroup$

Say regression as a model, overfitting means adding bunch of independent variables to explain a dependent variable. Adding even one independent variable, will increase the the fitting ability of the model. However, there are several problems/issues to doing this and some remedies to it.

Problems:

  1. The addition of the new independent variable does not significantly improve the model fitting.
  2. The new variable has correlation with other independent variables.
  3. The new variable has little to no real (theoretical) connection to the dependent variable.

Remedies:

  1. Check the significance of the variable. One can use Partial F-test, likelihood ratio test, $AIC$, $BIC$, etc.
  2. Multicollinearity can be detected and solved using variance inflation factor, principal component, etc.
  3. You can use step-wise regression to include or exclude the variable.
$\endgroup$
2
  • $\begingroup$ Thanks for your answer and listing the problems with overfitting and remedies to avoid it. $\endgroup$
    – SR1
    Jul 27, 2022 at 2:15
  • $\begingroup$ What is common is that as soon as anyone mentions any of the explicit trade-off criteria (AIC, BIC, etc.) then someone (even the same expert) asserts that it often gives the wrong answer, namely that it indulges or inhibits in the wrong direction. What is easier to agree is that goodness of fit and simplicity of fit are often in tension. $\endgroup$
    – Nick Cox
    Jul 27, 2022 at 10:05
0
$\begingroup$

I like this simple mathematical explanation for the regression case so I'll leave it here:

Overfitting is a consequence of an imbalance between the following three factors:

  • Number of training samples N. If you have many training samples, you should learn well, even if the model is complex. Conversely, if the model is complex but does not have enough training samples, you will overfit it.
  • Model order d. This refers to the complexity of the model (d <= N). If your training set is too small, you need to use a less complex model.
  • Noise variance σ2. This refers to the variance of the error added to the data. If σ increases, it is inevitable that the fitting will become more difficult. Hence it would require more training samples, and perhaps a less complex model would work better.

In short:

MSEtrain = σ2 (1 - d/N)

MSEtest = σ2 (1 + d/N)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.