7
$\begingroup$

Many statistics textbooks state that adding more terms into a linear model always reduces the sum of squares and in turn increases the r-squared value. This has led to the use of the adjusted r-squared. But is it possible that adding a term into a linear model reduces the sum of squares by zero and in turns keep the r squared value exactly the same?

$\endgroup$
3
  • 2
    $\begingroup$ The correct statement about $R^2$ is that adding new parameters does not reduce it. It doesn't have to be a strict increase. $\endgroup$
    – Aksakal
    Jan 12 '15 at 14:03
  • $\begingroup$ @Aksakal have you worded that correctly? $\endgroup$
    – luciano
    Jan 12 '15 at 14:05
  • $\begingroup$ See the explanation here. They call it nondecreasing property $\endgroup$
    – Aksakal
    Jan 12 '15 at 14:08
4
$\begingroup$

Certainly this can happen: if the new predictor is contained in the linear span of the predictors already in the model.

Think about it geometrically: your new "fitting subspace" (the possible linear combinations of your predictors) is exactly the same as the old one, so the optimal fit and the sum of squares is unchanged.

However, this is only a sufficient condition for $R^2$ to be unchanged, not a necessary one. Consider three points like this:

xx <- c(-1,0,1)
yy <- c(1,-2,1)
plot(xx,yy,pch=19)
abline(h=0)
abline(v=0)

model.1 <- lm(yy~1)
abline(model.1,col="red",lty=2)
summary(model.1)

model.2 <- lm(yy~xx)
abline(model.2,col="green",lty=3)
summary(model.2)

example

If we add xx as a predictor to the simple mean model, we get the same fit and the same $R^2$. Such a construction should be possible with larger models, as well.

$\endgroup$
8
  • 1
    $\begingroup$ But if the predictor has different values to predictors already in the model but absolutely no relationship with the response variable, will the r-squared stay the same? $\endgroup$
    – luciano
    Jan 12 '15 at 11:42
  • $\begingroup$ You may need to quantify "no" here. $\endgroup$ Jan 12 '15 at 11:44
  • $\begingroup$ I mean , no = zero $\endgroup$
    – luciano
    Jan 12 '15 at 11:47
  • $\begingroup$ I changed my answer, I was wrong about the "necessary" part. I'd assume that there is some geometric interpretation of updating OLS with a new predictor (scalar products between the new predictor and residuals of the smaller model?) that quantifies your "no" in a way that $R^2$ would be unchanged when adding a predictor. $\endgroup$ Jan 12 '15 at 12:10
  • 1
    $\begingroup$ The point is that even in an undetermined system, the betas may be different, but the fits $\hat{y}_i$ will be the same, and therefore also the $R^2$. $\endgroup$ Aug 17 '20 at 18:41
3
$\begingroup$

Adding more terms into a linear model may keep the r squared value exactly the same or increase the r squared value. It is called non-decreasing property of R square.

To demonstrate this property, first recall that the objective of least squares linear regression is $$ min{SSE}=min\displaystyle\sum\limits_{i=1}^n \left(e_i \right)^2= min_{\beta}\sum_{i=1}^n\left(y_i -\beta_0 - \beta_1x_{i,1} - \beta_2x_{i,2} -…- \beta_px_{i,p}\right)^2 $$ R square is $$ R^2=1-\frac{SSE}{SST} $$ When the extra variable is included, the objective of least squares linear regression becomes $$ min{SSE}=min_{\beta}\sum_{i=1}^n\left(y_i -\beta_0 - \beta_1x_{i,1} - \beta_2x_{i,2} -…- \beta_px_{i,p}-\beta_{p+1}x_{i,p+1}\right)^2 $$ If extra estimated coefficient($\beta_{p+1}$) is zero, the SSE and the R square will stay unchanged. Or if extra estimated coefficient($\beta_{p+1}$) takes a nonzero value , the SSE will reduce. In this case, the R square will increase, because it improves the quality of the fit.

$\endgroup$
9
  • $\begingroup$ I think your conclusion is not quite right, because it makes a critical unstated assumption: if the extra estimated coefficient is zero and if the estimates of the other coefficients do not change, then $R^2$ will remain the same. Also, it is possible for the extra estimated coefficient to be nonzero, yet the $R^2$ will not change. (This happens when the new variable lies within the span of the previous variables.) The logic of the demonstration is simpler than this: by setting $\hat\beta_{p+1}=0$ you will obtain the same solution as before, whence it is impossible to do any worse. $\endgroup$
    – whuber
    Apr 9 '19 at 21:24
  • $\begingroup$ @whuber When extra estimated coefficient is nonzero, the estimates of the other coefficients have to change to minimize the cost function. Cost function in linear regression is a convex function, so once parameters change, the cost function can not stay unchanged. If cost function change, it can only reduce. $\endgroup$ Apr 9 '19 at 21:57
  • 1
    $\begingroup$ I'm afraid that's not always the case. You implicitly assume the new variable does not lie in the span of the existing ones. When it does, the cost function is not strictly convex and adding the variable accomplishes nothing. $\endgroup$
    – whuber
    Apr 9 '19 at 21:59
  • $\begingroup$ @whuber When $\beta_{p+1}=\hat\beta_{p+1}\ne0$ is the solution for minJ(X), $\beta_{p+1}=0$ is not the solution for minJ(X). $\Rightarrow\frac{\partial{J(X)}}{\partial{\beta_{p+1}}}|_{\beta_{p+1}=0}\ne0\Rightarrow$ There must exist point around 0 which is less than $minJ(X)|_{\beta_{p+1}=0}$. $\Rightarrow J(X)|_{\beta_{p+1}=\hat\beta_{p+1}}<minJ(X)|_{\beta_{p+1}=0}= min\sum_{i=1}^n\left(y_i -\beta_0 - \beta_1x_{i,1} - \beta_2x_{i,2} -…- \beta_px_{i,p}\right)^2$ $\endgroup$ Apr 10 '19 at 0:07
  • 1
    $\begingroup$ @Iamanon Notice the numerator depends only on the residuals. That sum of squares of residuals is the squared distance from the response to the column space. This description makes it obvious that $SS_\text{res}$ does not depend on the actual values of the coefficients. $\endgroup$
    – whuber
    Aug 17 '20 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.