Does over fitting a model affect R Squared only or adjusted R Squared too?

Question

The total sum of squares for the variable being predicted is as the following: $$\mathrm{TSS}=\sum_{i=1}^{n}\left(y_{i}-\bar{y}\right)^2$$ and the residual sum of squares from the predictions from your model $$\mathrm{RSS}=\sum_{i=1}^{n}\left(y_{i}^{\,}-\hat{y}_i\right)^2$$ then: $$R^2 = 1-\frac{\mathrm{RSS}}{\mathrm{TSS}}$$

When $$R^2=1$$

We may have an over fitted model, and some of the features should be analyzed (using correlation) and to be later removed, and recalculate the $R^2$.

The adjusted $R^2$ is: $$adjR^2 =1 - (1-\frac{\mathrm{RSS}}{\mathrm{TSS}})*\frac{\mathrm{n-1}}{\mathrm{n-p-1}}$$

Where $n$ is the number of samples used, and $p$ is the number of features used.

When $p$ increases, the denominator decreases, which lead to the whole fraction to increase. Thus, $adjR^2$ decreases according the formula.

But isn't it that increasing the number of features of a model result an over fitting, which lead to inaccurate prediction? Or the overfitting only affect $R^2$?

Answer 1

$R^2$ is not primarily intended as a diagnostic for overfitting. In a nice straightforward linear model (no penalization of parameters, no model building, just a single pre-specified model etc.) it is meant to tell you what proportion of the variation in the data around the overall mean is "explained" by the model terms. I.e. if you have a single model that you wish to evaluate and if you are sure that there is no overfitting, then $R^2$ is a measure of how well your model explains individual variation.

Once there is a risk of overfitting due to too many model parameters relative to the amount of data $R^2$ is no longer reliable for this purpose (due to overfitting). The adjusted $R^2$ does attempt to account for that to some extent by penalizing the $R^2$ downwards away from 1. It is of course only a simple approximation. Both $R^2$ and adjusted $R^2$ suffer from the difficulty of looking at the model performance on the same data on which you fit a model (alternatives include e.g. cross-validation or the more simplistic validation set approach). But the adjusted $R^2$ should be less affected by overfitting than $R^2$ for the purpose of assessing how much variability the model truly "explains" (vs. how much it is overfitting noise). I guess to some extent the difference between $R^2$ and adjusted $R^2$ is in some sense an estimate of how much you are overfitting.

Once you consider more than one pre-specified single model or engage in model building (e.g. by deciding that features should be analyzed e.g. using correlation and be removed according to some criteria), both $R^2$ and adjusted $R^2$ will suffer from overfitting in ways that are not accounted for in either formula. In those scenarios you should not rely on them, at all.

Answer 2

$R^2$ always increases as you add additional parameters. It will never catch overfitting, unless you calculate $R^2$ on out-of-sample data.

Let's assume OLS regression for this example.

The loss function--what is being optimized--is the same as the residual sum of squares, and the smaller the residual sum of squares, the higher your $R^2$ value.

Let's examine the following model: $\hat{y} = \hat{\beta}_0 + \hat{\beta}_1x$. Say we get $R^2=0.75$ and find that unacceptably low, so we add a quadratic term to get $\hat{y} = \hat{\beta}_2 + \hat{\beta}_3x + \hat{\beta}_4x^2$. $R^2$ better not decrease. Otherwise, we would have a better model by just having $\hat{\beta}_4 = 0$, $\hat{\beta}_3 = \hat{\beta}_1$, and $\hat{\beta}_2 = \hat{\beta}_0$ (the same model as before but with an explicit zero coefficient on the quadratic term).