0
$\begingroup$

The total sum of squares for the variable being predicted is as the following: $$\mathrm{TSS}=\sum_{i=1}^{n}\left(y_{i}-\bar{y}\right)^2$$ and the residual sum of squares from the predictions from your model $$\mathrm{RSS}=\sum_{i=1}^{n}\left(y_{i}^{\,}-\hat{y}_i\right)^2$$ then: $$R^2 = 1-\frac{\mathrm{RSS}}{\mathrm{TSS}}$$

When $$R^2=1$$

We may have an over fitted model, and some of the features should be analyzed (using correlation) and to be later removed, and recalculate the $R^2$.

The adjusted $R^2$ is: $$adjR^2 =1 - (1-\frac{\mathrm{RSS}}{\mathrm{TSS}})*\frac{\mathrm{n-1}}{\mathrm{n-p-1}}$$

Where $n$ is the number of samples used, and $p$ is the number of features used.

When $p$ increases, the denominator decreases, which lead to the whole fraction to increase. Thus, $adjR^2$ decreases according the formula.

But isn't it that increasing the number of features of a model result an over fitting, which lead to inaccurate prediction? Or the overfitting only affect $R^2$?

$\endgroup$
  • $\begingroup$ I'm not sure I understand your question. First, increasing $p$ would lead to overfitting, which lead to inaccurate prediction... this is correct. Second, overfitting should not affect $R^2$. The more regressors (features) you have, the better you should be able to fit the training set. The effect on $adjR^2$, however, is ambiguous, since the expression is "penalized" by $p$ as you mentioned. $\endgroup$ – Art Sep 10 '19 at 8:07
  • $\begingroup$ The more regressors that are properly correlated with the output would not lead to overfitting right ? If I used 20 regressors from which 6 are dependent and should be removed, and having R squared equal 1 that is overfitting. But using 20 regressors where all of them are positivily correlated to the output, would lead to high value of R squared with no overfitting. That's what I need to understand if it is correct or not. @Art $\endgroup$ – alim1990 Sep 10 '19 at 8:13
3
$\begingroup$

$R^2$ is not primarily intended as a diagnostic for overfitting. In a nice straightforward linear model (no penalization of parameters, no model building, just a single pre-specified model etc.) it is meant to tell you what proportion of the variation in the data around the overall mean is "explained" by the model terms. I.e. if you have a single model that you wish to evaluate and if you are sure that there is no overfitting, then $R^2$ is a measure of how well your model explains individual variation.

Once there is a risk of overfitting due to too many model parameters relative to the amount of data $R^2$ is no longer reliable for this purpose (due to overfitting). The adjusted $R^2$ does attempt to account for that to some extent by penalizing the $R^2$ downwards away from 1. It is of course only a simple approximation. Both $R^2$ and adjusted $R^2$ suffer from the difficulty of looking at the model performance on the same data on which you fit a model (alternatives include e.g. cross-validation or the more simplistic validation set approach). But the adjusted $R^2$ should be less affected by overfitting than $R^2$ for the purpose of assessing how much variability the model truly "explains" (vs. how much it is overfitting noise). I guess to some extent the difference between $R^2$ and adjusted $R^2$ is in some sense an estimate of how much you are overfitting.

Once you consider more than one pre-specified single model or engage in model building (e.g. by deciding that features should be analyzed e.g. using correlation and be removed according to some criteria), both $R^2$ and adjusted $R^2$ will suffer from overfitting in ways that are not accounted for in either formula. In those scenarios you should not rely on them, at all.

$\endgroup$
  • $\begingroup$ So should we rely on mixed criterion ? $R^2$ , $adjR^2$, $p-values$, correlations, ...? $\endgroup$ – alim1990 Sep 11 '19 at 6:41
  • 1
    $\begingroup$ As mentioned: out of training sample methods are probably some of the better ideas (cross-validation, validation set etc.), if you want to know during training. If you then make decisions based on this, you probably want to reserve a final hold-out set for final unbiased evaluation. $\endgroup$ – Björn Sep 11 '19 at 12:30
1
$\begingroup$

$R^2$ always increases as you add additional parameters. It will never catch overfitting, unless you calculate $R^2$ on out-of-sample data.

Let's assume OLS regression for this example.

The loss function--what is being optimized--is the same as the residual sum of squares, and the smaller the residual sum of squares, the higher your $R^2$ value.

Let's examine the following model: $\hat{y} = \hat{\beta}_0 + \hat{\beta}_1x$. Say we get $R^2=0.75$ and find that unacceptably low, so we add a quadratic term to get $\hat{y} = \hat{\beta}_2 + \hat{\beta}_3x + \hat{\beta}_4x^2$. $R^2$ better not decrease. Otherwise, we would have a better model by just having $\hat{\beta}_4 = 0$, $\hat{\beta}_3 = \hat{\beta}_1$, and $\hat{\beta}_2 = \hat{\beta}_0$ (the same model as before but with an explicit zero coefficient on the quadratic term).

$\endgroup$
  • $\begingroup$ what do you mean by out-of-sample data? $\endgroup$ – alim1990 Sep 11 '19 at 5:42
  • $\begingroup$ @alim1990 You have 100 observations. Train your model on 80. Try it out on the remaining 20 that form your "out-of-sample" data set. This way, you can check if your model is capturing a real pattern or a coincidence in the data you happened to use to calculate the parameters. $\endgroup$ – Dave Sep 11 '19 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.