Difference between Adjusted R Squared and Predicted R Squared

Question

As we all know, depending solely on R squared to determine goodness of fit is faulty because the R squared value will always increase when more predictors are added, regardless of their physical correlation. With this in mind, many people recommend using the adjusted R squared value, especially if the equation will be used to estimate future values.

However, I recently stumbled upon something called the predicted R squared value, which relies on the predicted residual error sum of squares (PRESS) statistic. In this method,

• A data point from your dataset is removed
• A refitted linear regression model is generated
• The removed data point is plugged into the refitted linear model, generating a predicted value
• The removed data point is placed back into your dataset. Repeat from step 1 for the next data point until all data points have had a chance to be removed.

Afterward, you get a vector of predicted values which you subtract from their analogous true values to get a predicted residual value, which can be squared to convert it into a PRESS value. When calculating the predicted R Squared, the PRESS effectively replaces the sum of squares residuals value in the R Squared formula.

My question is which method is better at taking into account overmodeling, adjusted R Squared or predicted R Squared? The adjusted R Squared formula relies on the R squared value and the dataset size and predictor number, but the predicted R Squared completely re-calculates the sum of squares residual.

A part of a project I'm working on requires creating a multilinear regression model to correlate predictors (fictional example) such as grass length, intensity of green grass color, quantity of grass per square area, etc. (x variables) to cow happiness (y variable). The adjusted R squared and predicted R squared values react completely differently when I go from 3 to 4 to 5 predictors. The adjusted R squared value stays pretty much constant around 91% from 3 to 5 predictors. However, the predicted R squared value decreases from 87% to 71% to 60%. Which R Squared value applies more here?

Any help is appreciated. Thanks!

Answer 1

I won't attempt to give you a highly technical answer here.

I am inclined to trust the PRESS statistic more than I would trust adjusted R squared. The adjusted R squared is still an "in-sample" measure, while the PRESS is an "out-of-sample" measure. I would consider the out-of-sample measure to be more powerful generally.

Additionally, the adjusted R squared might not be very different from R squared if you have a small number of predictors compared to the number of observations you are fitting. So it might not be as informative as the PRESS statistic.

However, If what you are describing is correct, and the PRESS statistic is calculated correctly, then it is clear that the predictive performance of your model is suffering when you add predictors. But it is also not clear why your PRESS statistic is suffering so much compared to the adjusted R squared. It doesn't feel like that should be the case.

I would recommend checking that you are calculating the PRESS statistic correctly (if this is not something that is provided by the tool you are using to fit the model).

It is quite difficult to diagnose what is happening without having more detail of the issue at hand.

Answer 2

I will give a shot at a technical answer.

Let us assume that all regression assumptions are met and that the predictors have a multivariate normal distribution. In notation, we can summarize this as

$Y=\beta^{\top}X+\epsilon, \text{with} \quad X \sim N(\mu_X, \Sigma_X),\quad \epsilon \sim N(0,\sigma_\epsilon^2)$

with $Y$ representing the dependent variable and $X$ the independent variables/predictors, $\epsilon$ the error.

When using adjusted-R-squared, what we actually want to estimate is $$\rho^2=1-\frac{\sigma_\epsilon^2}{Var(Y)}.$$

In words: this is the amount of variance of the dependent variable that the best linear function f (as represented by the true regression weights $\beta$) explains. On a side note, I wrote a paper [1] that shows that often there are better ways to do this than standard adjusted R-squared.

For predicted R-squared, we almost use the same formula, only a different error term. Instead of estimating the irreducible error $\sigma_\epsilon^2$, we are interested in

$$E_{X,Y}([\hat{f}(X)-Y]^2)$$

where $\hat{f}$ is an estimate of the best function $f$ that we got from applying linear regression on a training set $D$.

Thus, the population value of predicted R-squared is $$\rho_c^2=1-\frac{E_{X,Y}([\hat{f}(X)-Y]^2)}{Var(Y)}$$ This is thus the amount of variance that the particular function $\hat{f}$ explains in the population.

We can decompose (this is the start of the well-known bias-variance decomposition from machine learning)

$$E([\hat{f}(X)-Y]^2)=E_{X,Y}([\hat{f}(X)-f(X)]^2)+E_{X,Y}([\hat{f}(X)-Y]^2)=E_{X,Y}([\hat{f}(X)-f(X)]^2)+\sigma_\epsilon^2.$$

Note that thus $E_{X,Y}((\hat{f}(X)-Y)^2) \geq\sigma_\epsilon^2$ and equality holds if $\hat{f}=f$, which is generally not the case.

In words: The error that the estimated function $\hat{f}$ makes consists of the difference between the estimated function $\hat{f}$ and the true function $f$ and the difference between the true function $f$ and the true value $Y$. Or, in other words, $\rho^2_c$ is an upper bound for $\rho^2$, and no function can predict better than $\sigma_\epsilon^2$.

So which measure is better at model selection? It depends. If you want to select the set of predictors that will lead to the most accurate predictions based on the current sample, then the predicted R-squared is better. If you want to select the set of predictors that will lead to the most accurate predictions if you had the whole population available (which is arguably often the question we would like to ask when we do explanatory modeling), then adjusted R-squared is better.

One word of warning. Predictive R-squared estimation via cross-validation works with rather minimal assumptions (only independence is needed). In contrast, adjusted R-squared estimation generally relies on all regression assumptions and the assumption that the predictors are multivariate normal.

References

$[1]$: Karch J (2020): Improving on adjusted R-squared. Collabra: Psychology (2020) 6 (1): 45. (link)