3
$\begingroup$

Many statistics textbooks state that adding more terms into a linear model always reduces the sum of squares and in turn increases the r-squared value. This has led to the use of the adjusted r-squared. But is it possible that adding a term into a linear model reduces the sum of squares by zero and in turns keep the r squared value exactly the same?

$\endgroup$
  • 2
    $\begingroup$ The correct statement about $R^2$ is that adding new parameters does not reduce it. It doesn't have to be a strict increase. $\endgroup$ – Aksakal Jan 12 '15 at 14:03
  • $\begingroup$ @Aksakal have you worded that correctly? $\endgroup$ – luciano Jan 12 '15 at 14:05
  • $\begingroup$ See the explanation here. They call it nondecreasing property $\endgroup$ – Aksakal Jan 12 '15 at 14:08
4
$\begingroup$

Certainly this can happen: if the new predictor is contained in the linear span of the predictors already in the model.

Think about it geometrically: your new "fitting subspace" (the possible linear combinations of your predictors) is exactly the same as the old one, so the optimal fit and the sum of squares is unchanged.

However, this is only a sufficient condition for $R^2$ to be unchanged, not a necessary one. Consider three points like this:

xx <- c(-1,0,1)
yy <- c(1,-2,1)
plot(xx,yy,pch=19)
abline(h=0)
abline(v=0)

model.1 <- lm(yy~1)
abline(model.1,col="red",lty=2)
summary(model.1)

model.2 <- lm(yy~xx)
abline(model.2,col="green",lty=3)
summary(model.2)

example

If we add xx as a predictor to the simple mean model, we get the same fit and the same $R^2$. Such a construction should be possible with larger models, as well.

$\endgroup$
  • 1
    $\begingroup$ But if the predictor has different values to predictors already in the model but absolutely no relationship with the response variable, will the r-squared stay the same? $\endgroup$ – luciano Jan 12 '15 at 11:42
  • $\begingroup$ You may need to quantify "no" here. $\endgroup$ – Stephan Kolassa Jan 12 '15 at 11:44
  • $\begingroup$ I mean , no = zero $\endgroup$ – luciano Jan 12 '15 at 11:47
  • $\begingroup$ I changed my answer, I was wrong about the "necessary" part. I'd assume that there is some geometric interpretation of updating OLS with a new predictor (scalar products between the new predictor and residuals of the smaller model?) that quantifies your "no" in a way that $R^2$ would be unchanged when adding a predictor. $\endgroup$ – Stephan Kolassa Jan 12 '15 at 12:10
0
$\begingroup$

Adding more terms into a linear model may keep the r squared value exactly the same or increase the r squared value. It is called non-decreasing property of R square.

To demonstrate this property, first recall that the objective of least squares linear regression is $$ min{SSE}=min\displaystyle\sum\limits_{i=1}^n \left(e_i \right)^2= min_{\beta}\sum_{i=1}^n\left(y_i -\beta_0 - \beta_1x_{i,1} - \beta_2x_{i,2} -…- \beta_px_{i,p}\right)^2 $$ R square is $$ R^2=1-\frac{SSE}{SST} $$ When the extra variable is included, the objective of least squares linear regression becomes $$ min{SSE}=min_{\beta}\sum_{i=1}^n\left(y_i -\beta_0 - \beta_1x_{i,1} - \beta_2x_{i,2} -…- \beta_px_{i,p}-\beta_{p+1}x_{i,p+1}\right)^2 $$ If extra estimated coefficient($\beta_{p+1}$) is zero, the SSE and the R square will stay unchanged. Or if extra estimated coefficient($\beta_{p+1}$) takes a nonzero value , the SSE will reduce. In this case, the R square will increase, because it improves the quality of the fit.

$\endgroup$
  • $\begingroup$ I think your conclusion is not quite right, because it makes a critical unstated assumption: if the extra estimated coefficient is zero and if the estimates of the other coefficients do not change, then $R^2$ will remain the same. Also, it is possible for the extra estimated coefficient to be nonzero, yet the $R^2$ will not change. (This happens when the new variable lies within the span of the previous variables.) The logic of the demonstration is simpler than this: by setting $\hat\beta_{p+1}=0$ you will obtain the same solution as before, whence it is impossible to do any worse. $\endgroup$ – whuber Apr 9 at 21:24
  • $\begingroup$ @whuber When extra estimated coefficient is nonzero, the estimates of the other coefficients have to change to minimize the cost function. Cost function in linear regression is a convex function, so once parameters change, the cost function can not stay unchanged. If cost function change, it can only reduce. $\endgroup$ – DavidCruise Apr 9 at 21:57
  • $\begingroup$ I'm afraid that's not always the case. You implicitly assume the new variable does not lie in the span of the existing ones. When it does, the cost function is not strictly convex and adding the variable accomplishes nothing. $\endgroup$ – whuber Apr 9 at 21:59
  • $\begingroup$ @whuber When $\beta_{p+1}=\hat\beta_{p+1}\ne0$ is the solution for minJ(X), $\beta_{p+1}=0$ is not the solution for minJ(X). $\Rightarrow\frac{\partial{J(X)}}{\partial{\beta_{p+1}}}|_{\beta_{p+1}=0}\ne0\Rightarrow$ There must exist point around 0 which is less than $minJ(X)|_{\beta_{p+1}=0}$. $\Rightarrow J(X)|_{\beta_{p+1}=\hat\beta_{p+1}}<minJ(X)|_{\beta_{p+1}=0}= min\sum_{i=1}^n\left(y_i -\beta_0 - \beta_1x_{i,1} - \beta_2x_{i,2} -…- \beta_px_{i,p}\right)^2$ $\endgroup$ – DavidCruise Apr 10 at 0:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.