10
$\begingroup$

I just started to learn about the following statistical measures, r-squared and adjusted r-squared and was wondering why can't we use adjusted r-squared for every regression model considering the fact that it penalizes the model for useless variables, unlike the former. Is there any advantage of r-squared over adjusted r-squared in some conditions?

$\endgroup$
2
  • 2
    $\begingroup$ Welcome to CV, Ronight. My answer here may prove of interest: stats.stackexchange.com/questions/336690/… $\endgroup$ – Alexis Jun 29 '18 at 18:29
  • $\begingroup$ Which ist better: a hammer or a banana? It depends on whether you need to hit a nail or are hungry. $R^2$ is good if you care about the variance explained, $adj. R^2$ is good if you want to know, if an extra predictor is worth it. $\endgroup$ – Bernhard Apr 6 at 12:05
5
$\begingroup$

Adjusted $R^2$ is the better model when you compare models that have a different amount of variables.

The logic behind it is, that $R^2$ always increases when the number of variables increases. Meaning that even if you add a useless variable to you model, your $R^2$ will still increase. To balance that out, you should always compare models with different number of independent variables with adjusted $R^2$.

Adjusted $R^2$ only increases if the new variable improves the model more than would be expected by chance.

$\endgroup$
5
  • $\begingroup$ Yes. I know how both r2 and adjusted r2 are calculated. My question was: should I use adjusted r2 for every regression model, considering the fact that it is better than r2? $\endgroup$ – Ronith Jun 29 '18 at 18:39
  • $\begingroup$ It's not about being better or worse. When you compare models use adjusted R2. When you only look at one model report R2, as it is the not adjusted measure of how much variance is explained by your model. $\endgroup$ – LN_P Jun 29 '18 at 18:45
  • $\begingroup$ Considering I have only one model and I use R2, it won't give me an indication of the insignificant variables I am adding to my model, in such a case why should I prefer R2 over adjusted R2? $\endgroup$ – Ronith Jun 29 '18 at 18:47
  • 1
    $\begingroup$ Neither R2 nor adjusted R2 will give you information about the significance of your variables. you have to look at your model output. When you want to see if some of your variables are insignificant when adding or removing them, then again you are comparing two models. Then you should use adjusted R2, because you are comparing models with a different number of independent variables in it. But if you only have one model , adjusted R2 will not tell you anythign about significant variables, as you don't compare it to another model... $\endgroup$ – LN_P Jun 29 '18 at 19:24
  • 1
    $\begingroup$ My bad! I should've framed my question correctly. What I meant was "If I use R2, it does not account for useless variables whereas adjusted R2 does. So why shouldn't I use adjusted R2 even if I have one model?" $\endgroup$ – Ronith Jun 29 '18 at 20:31
2
$\begingroup$

Before trying to answer the question, we need to clarify at least two things: "What do we mean by adjusted $R^2$? and "What do we mean by 'better'"?

Ultimately, the goal is to estimate the true proportion of variance explained in the population $\rho^2$. A multitude of different estimators were proposed. Gwowen Shieh$^{[1]}$ compares no less than 18 different estimators and Karch$^{[3]}$ compares 20 different estimators. A good overview can be found in $[2]$ and $[3]$. So the term "adjusted $R^2$" is ambiguous. It's no help that different software implement different formulas, making it difficult to compare the "adjusted $R^2$". R, for example, uses the Wherry Formula-1 according to the nomenclature of Yin et al.$^{[2]}$ (see here).

Now what do we mean by "better"? Shieh$^{[1]}$ and Karch$^{[3]}$ used simulations to compare the different estimators with respect to bias and mean squared error (MSE). Depending on what researches prioritize, what's the "best" estimator varies.

For example, Karch$^{[3]}$ found that:

The exact Olkin-Pratt estimator was optimal. It was the only estimator that was unbiased across all conditions. Additionally, it had practically identical MSE within any condition compared to other unbiased estimators within that condition. Consequently, under this perspective, the exact Olkin-Pratt estimator should always be used.

But he also writes:

Only if the researcher is confident that minimizing MSE is more critical than unbiasedness should a different estimator be used. In this case, I recommend an individualized choice based on the strategy described at the beginning of this discussion and if this is not feasible the positive-part version of the Ezekiel estimator.

Interestingly, even the normal, unadjusted $R^2$ which is clearly positively biased had the lowest mean squared error in at least one simulation scenario.

References

$[1]$: Shieh G (2008): Improved shrinkage estimation of squared multiple correlation coefficient and squared cross-validity coefficient. Organizational Research Methods, 11(2): 387-407 (link)

$[2]$: Yin P, Fan X (2001): Estimating $R^2$ shrinkage in multiple regression: A comparison of different analytical methods. The Journal of Experimental Education, 69(2): 203-224 (link)

$[3]$: Karch J (2020): Improving on adjusted R-squared. Collabra: Psychology (2020) 6 (1): 45. (link)

$\endgroup$
1
0
$\begingroup$

Yes, there is an advantage to $R^2$: It has a direct interpretation as the proportion of variance in the dependent variable that is accounted for by the model. Adjusted $R^2$ does not have this interpretation.

Also, you write that adjusted $R^2$ "penalizes the model for useless variables". That is true but incomplete. First, almost no variable is totally useless. $R^2$ will increase even if we add random noise, because, just by chance, there will be some relationship (see below).

Second, adjusted $R^2$ lowers $R^2$ for every independent variable, useless or not. In fact, in some cases (see below) it works very badly for noise variables

set.seed(1234)  #Sets a seed


x1 <- rnorm(1000)  #Standard Normal, N = 1000
x2 <- rnorm(1000)  #Normal, N = 1000

y <- 3*x1 + rnorm(1000, 0, 4)  #No relation with X2

m1 <- lm (y~x1)
summary(m1) #R2 = 0.3627, adjusted = 0.3621

m2 <- lm (y~x1 + x2)
summary(m2) #R2 = 0.3635, adjusted = 0.3622

Note that m2 would be slightly preferred, even using adjusted $R^2$.

Note also that I didn't have to "fish around" for an example like this, this was the first one I tried.

Adjusted $R^2$ is much more useful for comparing models where all the IVs are useful. It's a way of adjusting for the complexity of a model.

$\endgroup$
6
  • $\begingroup$ I will argue that your example proves the utility of adjusted R-squared as you would need a very strong argument to chose M2 over M1 because M2 is more complex and requires an additional assumption (X2 is important) for minimal improvement in R-squared (adjusted or not). Further, adjusted R-squared can still be interpreted as R-squared raw, just with the caveat that a penalization has been applied: 'After adjusting for number of independent variables relative to the sample size, approximately Z% of observed variation in Y can be explained by the O-order regression model that utilizes X1-Xi.' $\endgroup$ – LSC Jan 4 '20 at 15:15
  • $\begingroup$ I feel that this answer is very misleading. Adjusted R2 is a much better estimate of explained variance. Random variables don’t explain anything. It is just hallucination under extreme bias. $\endgroup$ – Cagdas Ozgenc Jan 4 '20 at 15:17
  • $\begingroup$ @LSC Interesting. Would you argue that the adjusted R-squared has more interpretive value? Should we default to the raw r-squared in cases where we are not doing any model comparison? $\endgroup$ – Thomas Bilach Feb 28 '20 at 14:18
  • $\begingroup$ @Tom, what do you mean by "more interpretive"? I think even without model comparisons R-squared adjusted is helpful in a multiple regression (>1 independent variable) context because it allows you to see the explained variation in the DV after penalization for # of terms in the model. It is also helpful when not comparing models to compare the unadjusted R-squared with the adjusted R-squared (again when # predictors/non-intercept terms>1) because you can see how the R-squared (raw) is inflated due to extra/superfluous terms in the model. $\endgroup$ – LSC Feb 28 '20 at 23:11
  • $\begingroup$ @LSC I should have been more clear. In other words, should the adjusted R-squared take precedence over the unadjusted R-squared in terms of reporting in scientific journals, working papers, etc.? $\endgroup$ – Thomas Bilach Feb 28 '20 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.