2
$\begingroup$

Let's consider a multiple linear regression formula:

$ \hat{y} = \beta_0 + \beta_1 \hat{x}_1 + \beta_2 \hat{x}_2 $ (1)

which produces adjusted $R^2 = r_1$.

Now I want to add to one predictor to the (1) which turns into:

$ \hat{y} = \beta_0 + \beta_1 \hat{x}_1 + \beta_2 \hat{x}_2 + \beta_3 \hat{x}_3 $ (2)

which produces adjusted $R^2 = r_2$.

If the data fed into (1) and (2) are exactly the same, is there any way to explain $r_2 < r_1$ apart from a code bug?

$\endgroup$
  • 2
    $\begingroup$ i think you might be confusing {R-sq} and {adjusted R-sq} here. $\endgroup$ – charles Feb 19 '14 at 2:02
  • 1
    $\begingroup$ @charles indeed $\endgroup$ – Francesco Feb 19 '14 at 2:31
  • 1
    $\begingroup$ The very large number of threads on our site discussing adjusted R-squared are worth reviewing in this context. $\endgroup$ – whuber Feb 19 '14 at 15:24
11
$\begingroup$

Yes, it's definitely possible for adjusted $R^2$ to decrease when you add parameters.

Ordinary $R^2$ can't decrease, but adjusted-$R^2$ certainly can. We can write the relationship between the two like so:

$R_{adj}^2 = R^2-(1-R^2)\frac{p}{n-p-1}$

Note that both terms in the product $(1-R^2)\cdot\frac{p}{n-p-1}$ are positive (unless $R^2=1$), so if $R^2<1$, $R_{adj}^2 < R^2$.

If $R^2<\frac{p}{n-1}$, then adjusted-$R^2$ will be negative.

$R_{adj}^2$ will decrease if the $R^2$ for a model with an additional term if the second model's $R^2$ didn't increase from that for the first model by at least as much as would be expected for an unrelated variable.

We can see this happen quite easily: I just generated three unrelated variables in R (via)
x1=rnorm(20);x2=rnorm(20);y=rnorm(20)

  1. If we fit a linear regression with just the first $x$ (lm(y~x1)), the adjusted $R^2$ is smaller than with the null model (which is 0):
    Multiple R-squared: 0.0007048, Adjusted R-squared: -0.05481

  2. If we fit both independent variables (lm(y~x1+x2)), the adjusted $R^2$ goes down again (and the $R^2$ – necessarily – goes up):
    Multiple R-squared: 0.00199, Adjusted R-squared: -0.1154

For adjusted $R^2$ to increase, its addition has to explain more additional variation in the data than would be expected from an unrelated variable; it's possible for an unrelated variable to add less than it would be expected to, just by chance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.