2
$\begingroup$

I understand from this question - When is R squared negative?

that the R squared value of a linear regression model can be negative if the intercept is constrained. And this makes sense if you define R squared as -

$$R^2 = 1-\frac{SSE}{SST}$$

One says $SSE>SST$. But then, $$SST = SSA + SSE $$ Total Sum of squares = Sum of squared errors + Sum of squared residual. And with this we get - $$R^2 = \frac{SSA}{SST}$$ And now it is hard to imagine how $R^2$ can be negative. Aren't SSA and SST >0 always?

$\endgroup$
2
$\begingroup$

I found a proof for SST=SSE+SSR in a different thread (link below). The proof relies on two equations derived from least square regression. If your regression was not not based on LS, those conditions would be violated and the equation SST=SSE+SSR wouldn't hold, which removes the contradiction of a negative R-squared.

https://math.stackexchange.com/questions/709419/prove-sst-ssessr

$\endgroup$
  • $\begingroup$ This is a good find, but reproducing the result here (with credit to the author) would probably garner you many upvotes. $\endgroup$ – Sycorax Dec 7 '15 at 19:40
  • $\begingroup$ Oh well, there are more important things in life than upvotes. $\endgroup$ – Jessica Dec 7 '15 at 20:26
2
$\begingroup$

$$ \begin{eqnarray*} SST&=&\sum_{i=1}^n (y_i-\bar y)^2\\&=&\sum_{i=1}^n (y_i-\hat y_i+\hat y_i-\bar y)^2\\&=&\sum_{i=1}^n (y_i-\hat y_i)^2+2\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)+\sum_{i=1}^n(\hat y_i-\bar y)^2\\&=&SSE+SSR+2\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y) \end{eqnarray*} $$ When an intercept($\beta_0$) is included in linear regression, $$\hat{y}_i = \beta_0 + \beta_1x_{i,1} + \beta_2x_{i,2} +…+ \beta_px_{i,p}$$ It can be proved that $\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)=0$ here

So $$ SST=SSE+SSR+2\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)=SSE+SSR $$

$$ R^2 = 1-\frac{SSE}{SST}=\frac{SSR}{SST}\geqslant0 $$ However, when an intercept is not included in linear regression, $\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)\neq0$

If $\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)<0$, $SST=SSE+SSR+2\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)$may be less than $SSE$

In this case, $$ \frac{SSE}{SST}>1 $$ $$ R^2 = 1-\frac{SSE}{SST}<0 $$

$\endgroup$
1
$\begingroup$

The point in the accepted answer to your linked question illustrates the answer to this.

If the regression model is so bad (for example because of a foolishly constrained intercept, but there are other potential causes) that a constant predictor (i.e. a horizontal line, with an intercept equal to the mean of the observed data) would fit the observed data better, then the sum of squares of the errors will exceed the sum of squares calculated from the original data (of the difference from the mean), and you will find $R^2$ going negative.

In such a case, you might want to think about whether the expression $SST = SSA + SSE$ is meaningful when the model is so bad that $SSE \gt SST$ and $SST$ represents the sum of squares calculated from the original data.

Unconstrained simple linear regression avoids this problem, since it just minimises $SSE$, and so $SST$ is an upper bound on $SSE$

$\endgroup$
  • $\begingroup$ SST = SSA + SSE should hold irrespective of the way linear regression has been fit and it should hold for all lines (not the best fitting one) shouldn't it? $\endgroup$ – ryu576 Feb 12 '14 at 8:35
  • $\begingroup$ @ryu576: Not if you can calculate SST before you do the regression (it is related to the variance of the dependent variable in the data). $\endgroup$ – Henry Feb 12 '14 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.