Let's say I have two regression models

(I) $y_t=\beta_1+\beta_2 x_2+u_t$

(II) $y_t=\beta_1+\beta_2 x_2+\beta_3 x_3 + u_t$

How the omission of relevant variable (not irrelevant variable) affects adjusted $R^2$?

That's when I compare Adj-$R^2$ for two models, what can I say? Which one less?

I see this while studying lecture notes. So I Try to understand the topic. Please explain this. Thank you:)

  • $\begingroup$ Can you tell me why we normally prefer to use an Adj-$R^2$ rather than the $R^2$? Hint: write down the formula for the Adj-$R^2$ and see what it has more than $R^2$. $\endgroup$
    – Stat
    Apr 4 '14 at 22:19
  • $\begingroup$ Because adjusted one inçlerden penalty. That's when we add irrelevant variable, adjusted one decreases but $R^2$ increases. So we prefer adjusted. @Stat but when the relevant variable is omitted, I am confused. $\endgroup$
    – 1190
    Apr 4 '14 at 22:36
  • $\begingroup$ OK here is the problem. You said that "when we add irrelevant variable ... $R^2$ increases. This is true. But what is the reason for that increase? Is this because the variable was irrelevant or is it because when you add a variable (whether relevant or irrelevant), you are actually having a more complex model (with more parameters) that reduces your $SSE$? Which one? $\endgroup$
    – Stat
    Apr 4 '14 at 22:57
  • $\begingroup$ Adding irrelevant variable increases SSE. I am begining level. So I have No enough knowledge to answer your questions properly. @Stat $\endgroup$
    – 1190
    Apr 5 '14 at 0:05

As described in your models, if x3 is a relevant variable Model II will have both a higher R Square and higher Adjusted R Square than Model I. Also, Model II will have a lower Standard Error than Model I. Thus, you should keep this x3 variable and chose Model II.

If x3 is not a relevant variable Model II will have a higher R Square, but a lower Adjusted R Square than Model I. It also most probably will have a higher Standard Error than Model I. In this case, you should exclude the x3 variable and stick with Model I.

That's kind of the basics. In reality, once you add a few variables the added explanatory power of adding additional variables increasingly diminishes. That's even though those variables are deemed relevant and that your Adjusted R Square keeps on rising. However, let's say that adding x3 would cause your Adjusted R Square to increase by 0.15; that's a lot, and you would definitely keep x3. Now, you add another variable x4. And, the resulting Adjusted R Square increases by only 0.03. I think many people would not add x4. It may not be that worth it. Adding it may lead to a model that is overfit. You can test whether a model is overfit by using a Hold Out sample. The latter is probably more important than the ultimate level of your Adjusted R Square.

Going back to your two models (I and II). You should actually test both of them to check their performance in a Hold Out sample. Only after doing so, can you be sure that Model II is better instead of simply being overfit (which the Adjusted R Square will not capture that).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.