Which is better: r-squared or adjusted r-squared?

Question

I just started to learn about the following statistical measures, r-squared and adjusted r-squared and was wondering why can't we use adjusted r-squared for every regression model considering the fact that it penalizes the model for useless variables, unlike the former. Is there any advantage of r-squared over adjusted r-squared in some conditions?

Answer 1

Before trying to answer the question, we need to clarify at least two things: "What do we mean by adjusted $R^2$? and "What do we mean by 'better'"?

Ultimately, the goal is to estimate the true proportion of variance explained in the population $\rho^2$. A multitude of different estimators were proposed. Gwowen Shieh$^{[1]}$ compares no less than 18 different estimators and Karch$^{[3]}$ compares 20 different estimators. A good overview can be found in $[2]$ and $[3]$. So the term "adjusted $R^2$" is ambiguous. It's no help that different software implement different formulas, making it difficult to compare the "adjusted $R^2$". R, for example, uses the Wherry Formula-1 according to the nomenclature of Yin et al.$^{[2]}$ (see here).

Now what do we mean by "better"? Shieh$^{[1]}$ and Karch$^{[3]}$ used simulations to compare the different estimators with respect to bias and mean squared error (MSE). Depending on what researches prioritize, what's the "best" estimator varies.

For example, Karch$^{[3]}$ found that:

The exact Olkin-Pratt estimator was optimal. It was the only estimator that was unbiased across all conditions. Additionally, it had practically identical MSE within any condition compared to other unbiased estimators within that condition. Consequently, under this perspective, the exact Olkin-Pratt estimator should always be used.

But he also writes:

Only if the researcher is confident that minimizing MSE is more critical than unbiasedness should a different estimator be used. In this case, I recommend an individualized choice based on the strategy described at the beginning of this discussion and if this is not feasible the positive-part version of the Ezekiel estimator.

Interestingly, even the normal, unadjusted $R^2$ which is clearly positively biased had the lowest mean squared error in at least one simulation scenario.

References

$[1]$: Shieh G (2008): Improved shrinkage estimation of squared multiple correlation coefficient and squared cross-validity coefficient. Organizational Research Methods, 11(2): 387-407 (link)

$[2]$: Yin P, Fan X (2001): Estimating $R^2$ shrinkage in multiple regression: A comparison of different analytical methods. The Journal of Experimental Education, 69(2): 203-224 (link)

$[3]$: Karch J (2020): Improving on adjusted R-squared. Collabra: Psychology (2020) 6 (1): 45. (link)

Answer 2

Adjusted $R^2$ is the better model when you compare models that have a different amount of variables.

The logic behind it is, that $R^2$ always increases when the number of variables increases. Meaning that even if you add a useless variable to you model, your $R^2$ will still increase. To balance that out, you should always compare models with different number of independent variables with adjusted $R^2$.

Adjusted $R^2$ only increases if the new variable improves the model more than would be expected by chance.

Answer 3

Yes, there is an advantage to $R^2$: It has a direct interpretation as the proportion of variance in the dependent variable that is accounted for by the model. Adjusted $R^2$ does not have this interpretation.

Also, you write that adjusted $R^2$ "penalizes the model for useless variables". That is true but incomplete. First, almost no variable is totally useless. $R^2$ will increase even if we add random noise, because, just by chance, there will be some relationship (see below).

Second, adjusted $R^2$ lowers $R^2$ for every independent variable, useless or not. In fact, in some cases (see below) it works very badly for noise variables

set.seed(1234)  #Sets a seed


x1 <- rnorm(1000)  #Standard Normal, N = 1000
x2 <- rnorm(1000)  #Normal, N = 1000

y <- 3*x1 + rnorm(1000, 0, 4)  #No relation with X2

m1 <- lm (y~x1)
summary(m1) #R2 = 0.3627, adjusted = 0.3621

m2 <- lm (y~x1 + x2)
summary(m2) #R2 = 0.3635, adjusted = 0.3622

Note that m2 would be slightly preferred, even using adjusted $R^2$.

Note also that I didn't have to "fish around" for an example like this, this was the first one I tried.

Adjusted $R^2$ is much more useful for comparing models where all the IVs are useful. It's a way of adjusting for the complexity of a model.