0
$\begingroup$

As stated above, How does one show mathematically that adding many variables in a linear regression will only increase R-squared ?

$\endgroup$
4
  • $\begingroup$ Add orthogonal variables will not improve R-squared. $\endgroup$ – SmallChess Mar 26 '17 at 13:16
  • 1
    $\begingroup$ You would be correct if you say that it will increase $R^2$ rather than "give a very high value for R-squared. $\endgroup$ – Michael R. Chernick Mar 26 '17 at 13:53
  • 1
    $\begingroup$ @MichaelChernick Not exactly. $R^2$ might not increase. $\endgroup$ – SmallChess Mar 26 '17 at 14:28
  • $\begingroup$ You are right Student T. I should have said that it will not decrease $R^2$. . $\endgroup$ – Michael R. Chernick Mar 26 '17 at 16:33
3
$\begingroup$

This is a general property of nested models: When you add parameters to a model (retaining all previous parameters) the fit cannot get worse because merely setting the new parameters to zero makes the expanded model identical to the nested (restricted) model. Notice that the additional parameter might not make the fit improve, but usually will because by chance the additional complexity of the expanded model will at least better fit random noise in the data.

In the specific case of linear regression, the model $\hat{y} = \beta_0 +\beta_1 x_1 + \beta_2 x_2$ must fit the data at least as well as the restricted model $\hat{y} = \beta_0 +\beta_1 x_1 + 0 x_2$ where $\beta_2 = 0$. Or, argue it this way: Let $\beta_0^*$, $\beta_1^*$, and $\beta_2^*$ be the least-squares estimates. Then any other value of $\beta_2$ must make $R^2$ worse; in particular $\beta_2=0$ must make $R^2$ worse (unless $\beta_2^*$ happens already to be exactly $0$).

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.