4
$\begingroup$

I saw this answer (https://stats.stackexchange.com/a/6374/27969) that said that $R^2$ can be negative if it's defined as $1-\frac{SSR}{SST}$. SSR is the sum of squared residuals which is defined as

\begin{align} SSR &= \displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_{i})^2 \\ &= \displaystyle\sum\limits_{i=1}^n (y_i - \hat{\beta_1} x_{i1} - \hat{\beta_2} x_{i2} - \ldots - \hat{\beta_k} x_{ik} )^2 \\ \end{align}

when theres no intercept, and SST is the total sum of squares, or

\begin{align} SST &= \displaystyle\sum\limits_{i=1}^n (y_i - \bar{y})^2 \\ \end{align}

Also SSE, or the explained sum of squares, is

\begin{align} SSE &= \displaystyle\sum\limits_{i=1}^n (\bar{y} - \hat{y}_i)^2 \end{align}

in the notation of my book. I just got to the section in my textbook that mentions this too and I'm confused. Why is this true?

Here's what I could work out. I know that

\begin{align} R^2 &= 1-\frac{SSR}{SST} \\ \end{align}

So for $R^2 < 0$, we need $\frac{SSR}{SST} > 1$. Here's what I worked out:

\begin{align} SSR &> SST \\ \displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_{i})^2 &> \displaystyle\sum\limits_{i=1}^n (y_i - \bar{y}_{i})^2 \\ \end{align}

Here's the proof I tried to work out:

\begin{align} SSR &> SST \\ SSR &> \displaystyle\sum\limits_{i=1}^n (y_i - \bar{y})^2 \\ &> \displaystyle\sum\limits_{i=1}^n \left[ (y_i - \hat{y}_i) - (\bar{y} - \hat{y}_i)\right]^2 \\ &> \displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i)^2 - 2\displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i)(\bar{y} - \hat{y}_i) - \displaystyle\sum\limits_{i=1}^n (\bar{y}_i - \hat{y}_i)^2 \\ &> SSR - 2\bar{y}\displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i) + 2\displaystyle\sum\limits_{i=1}^n \hat{y}_i(y_i - \hat{y}_i) - \displaystyle\sum\limits_{i=1}^n (\bar{y} - \hat{y}_i)^2 \\ &> SSR - 2\bar{y}\displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i) + 2\displaystyle\sum\limits_{i=1}^n \hat{y}_i(y_i - \hat{y}_i) - SSE \\ 0 &> -2\bar{y}\displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i) + 2\displaystyle\sum\limits_{i=1}^n \hat{y}_i(y_i - \hat{y}_i) - SSE \\ SSE &> -2\bar{y}\displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i) + 2\displaystyle\sum\limits_{i=1}^n \hat{y}_i(y_i - \hat{y}_i) \\ \end{align}

This is where I get stuck. Is the proof on the right path, or is there a mistake before I got stuck? Hints would be great.

I'm starting to get why this is the case logically. SSR is the sum of the squares of the distances from the points to the regression line, while SST is the sum of the squares of the distances from the points to a line drawn through $\bar{y}$. $R^2$ is negative because $SSR > SST$, which means that because we impose the limitation that the regression line MUST pass through the origin, our regression line is an even worse fit than if we just drew a horizontal line at $y =\bar{y}$ and used that. The distances from the points to the regression line are higher than the distances from the points to the line at $\bar{y}$ (ok thhe sum of squares of the distances is larger). Is that the right logic?

$\endgroup$
6
  • 1
    $\begingroup$ There are other discussions about the R-squared when there's no interecept, such as stats.stackexchange.com/questions/47527/… Did you take a look at these discussions too ? $\endgroup$ Jul 27, 2013 at 19:27
  • $\begingroup$ @StéphaneLaurent I guess I didn't understand why SST is the sum of squares around 0 in the model with no intercept (the other answer says that), since it's still defined the same way. I think my question is actually a stupid question because I won't be able to prove mathematically that $SSR > SST$ because even though it might be true, it's not always true. Is that right? $\endgroup$
    – M T
    Jul 27, 2013 at 19:34
  • 1
    $\begingroup$ To prove something can happen, you need only find an example, don't you? $\endgroup$
    – Glen_b
    Jul 28, 2013 at 0:53
  • $\begingroup$ @Glen_b yeah this was a stupid question I realize now. what do I do just delete it? $\endgroup$
    – M T
    Jul 28, 2013 at 1:31
  • $\begingroup$ @Michael I think the title question is a useful question; some of the body might need to change though. If you think you can answer the question (from the sound of it I think you could), you should do so. If not, you can remove the parts that seek to prove something you don't need to and leave it to be answered; the question should stay $\endgroup$
    – Glen_b
    Jul 28, 2013 at 14:07

3 Answers 3

3
$\begingroup$

As you said, the problem is that it is not always true that $R^2<0$ when there is no intercept and therefore you just need to find and example to show that $R^2$ may be less than 0.

I'm just proposing an extreme example where $SSR>SST$:

Residuals for y=1

In any sample with y=constant and $\bar x$ different than 0, $SSR>0$ (the squares of the blue lines in the graph) while $SST=0$. Please notice that if we considered an intercept, the regression line would be horizontal and $SSR$ would equal zero, too.

$\endgroup$
0
$\begingroup$

The role of an intercept in a regression is to capture the mean of errors $\varepsilon_i$, so that $E[\varepsilon_i]=0$ and $E[X\varepsilon_i]=0$. Here, we have $y=X\beta+\varepsilon$ or $y=c+\tilde X\beta+\varepsilon$, where $c$ is an intercept and $\tilde X$ - bona fide regressors. This is very convenient, because now $E[y]=c+E[\tilde X\beta]$.

This also allows to decompose the total sum of squares TTS as follows: $$var[y]=var[X\beta+\varepsilon]=var[X\beta]+var[\varepsilon]$$ Then you get $$E[y^2]=E[(X\beta)^2]+\sigma^2$$ because $E[y]=E[X\beta]$. This your R square equation: TTS = RSS + SSE.

When you remove the intercept, then the first two expectations are not zero, and the TTS doesn't add up the way you'd like.

$\endgroup$
-1
$\begingroup$

Recently I found on the LinkedIn the attached illustration with two examples. The thread was about two kinds of R2: centered, that is taught at school, and uncentered R2. It was shown, that the R statistical package switches to the latter for zero-intercept models, as this one is always positive.

Overall R2 does not have to be always lower than zero. It will be so only if the model predicts worse than just the global mean (and there you have no intercept - no global mean).

enter image description hereI have no rights to this picture Source: LinkedIn, posted by Adrian Olszewski, I have no rights to this picture, just saved it to my computer.

Yes, it is difficult to read, but there is a whole discussion on the LinkedIn, where mr Olszewski explains the way R handles this situation. I shared this because it gives two examples when the centered R2 takes negative value. This speaks to my mind when I see it. The points fluctuate at y=10, so the global mean is about so. Because we force the line to go through (0,0), the model performs much worse than the global mean and the R2 is -704. On the second picture the situation is less critical, and here R2 = -0.86. I know this is not an explanation but it contains two illustrations.

$\endgroup$
2
  • 2
    $\begingroup$ This image is much much too busy to be of use. $\endgroup$ Apr 21, 2021 at 22:28
  • $\begingroup$ Yes it is difficult to read, but there is a whole discussion on the LinkedIn, where mr Olszewski explains the way R handles this situation. I shared this because it gives two examples when the centered R2 takes negative value. This speaks to my mind when I see it. The points fluctuate at y=10, so the global mean is about so. Because we force the line to go through (0,0), the model performs much worse than the global mean and the R2 is -704. On the second picture the situation is less critical, and here R2 = -0.86. I know this is not an explanation but it contains two illustrations. $\endgroup$ Apr 21, 2021 at 22:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.