Why can $R^2$ be negative in a multiple regression that doesn't have an intercept ($\hat{\beta}_0 = 0$)? [duplicate]

Question

I saw this answer (https://stats.stackexchange.com/a/6374/27969) that said that $R^2$ can be negative if it's defined as $1-\frac{SSR}{SST}$. SSR is the sum of squared residuals which is defined as

\begin{align} SSR &= \displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_{i})^2 \\ &= \displaystyle\sum\limits_{i=1}^n (y_i - \hat{\beta_1} x_{i1} - \hat{\beta_2} x_{i2} - \ldots - \hat{\beta_k} x_{ik} )^2 \\ \end{align}

when theres no intercept, and SST is the total sum of squares, or

\begin{align} SST &= \displaystyle\sum\limits_{i=1}^n (y_i - \bar{y})^2 \\ \end{align}

Also SSE, or the explained sum of squares, is

\begin{align} SSE &= \displaystyle\sum\limits_{i=1}^n (\bar{y} - \hat{y}_i)^2 \end{align}

in the notation of my book. I just got to the section in my textbook that mentions this too and I'm confused. Why is this true?

Here's what I could work out. I know that

\begin{align} R^2 &= 1-\frac{SSR}{SST} \\ \end{align}

So for $R^2 < 0$, we need $\frac{SSR}{SST} > 1$. Here's what I worked out:

\begin{align} SSR &> SST \\ \displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_{i})^2 &> \displaystyle\sum\limits_{i=1}^n (y_i - \bar{y}_{i})^2 \\ \end{align}

Here's the proof I tried to work out:

\begin{align} SSR &> SST \\ SSR &> \displaystyle\sum\limits_{i=1}^n (y_i - \bar{y})^2 \\ &> \displaystyle\sum\limits_{i=1}^n \left[ (y_i - \hat{y}_i) - (\bar{y} - \hat{y}_i)\right]^2 \\ &> \displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i)^2 - 2\displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i)(\bar{y} - \hat{y}_i) - \displaystyle\sum\limits_{i=1}^n (\bar{y}_i - \hat{y}_i)^2 \\ &> SSR - 2\bar{y}\displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i) + 2\displaystyle\sum\limits_{i=1}^n \hat{y}_i(y_i - \hat{y}_i) - \displaystyle\sum\limits_{i=1}^n (\bar{y} - \hat{y}_i)^2 \\ &> SSR - 2\bar{y}\displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i) + 2\displaystyle\sum\limits_{i=1}^n \hat{y}_i(y_i - \hat{y}_i) - SSE \\ 0 &> -2\bar{y}\displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i) + 2\displaystyle\sum\limits_{i=1}^n \hat{y}_i(y_i - \hat{y}_i) - SSE \\ SSE &> -2\bar{y}\displaystyle\sum\limits_{i=1}^n (y_i - \hat{y}_i) + 2\displaystyle\sum\limits_{i=1}^n \hat{y}_i(y_i - \hat{y}_i) \\ \end{align}

This is where I get stuck. Is the proof on the right path, or is there a mistake before I got stuck? Hints would be great.

I'm starting to get why this is the case logically. SSR is the sum of the squares of the distances from the points to the regression line, while SST is the sum of the squares of the distances from the points to a line drawn through $\bar{y}$. $R^2$ is negative because $SSR > SST$, which means that because we impose the limitation that the regression line MUST pass through the origin, our regression line is an even worse fit than if we just drew a horizontal line at $y =\bar{y}$ and used that. The distances from the points to the regression line are higher than the distances from the points to the line at $\bar{y}$ (ok thhe sum of squares of the distances is larger). Is that the right logic?

Answer 1

As you said, the problem is that it is not always true that $R^2<0$ when there is no intercept and therefore you just need to find and example to show that $R^2$ may be less than 0.

I'm just proposing an extreme example where $SSR>SST$:

In any sample with y=constant and $\bar x$ different than 0, $SSR>0$ (the squares of the blue lines in the graph) while $SST=0$. Please notice that if we considered an intercept, the regression line would be horizontal and $SSR$ would equal zero, too.

Answer 2

The role of an intercept in a regression is to capture the mean of errors $\varepsilon_i$, so that $E[\varepsilon_i]=0$ and $E[X\varepsilon_i]=0$. Here, we have $y=X\beta+\varepsilon$ or $y=c+\tilde X\beta+\varepsilon$, where $c$ is an intercept and $\tilde X$ - bona fide regressors. This is very convenient, because now $E[y]=c+E[\tilde X\beta]$.

This also allows to decompose the total sum of squares TTS as follows: $$var[y]=var[X\beta+\varepsilon]=var[X\beta]+var[\varepsilon]$$ Then you get $$E[y^2]=E[(X\beta)^2]+\sigma^2$$ because $E[y]=E[X\beta]$. This your R square equation: TTS = RSS + SSE.

When you remove the intercept, then the first two expectations are not zero, and the TTS doesn't add up the way you'd like.

Answer 3

Recently I found on the LinkedIn the attached illustration with two examples. The thread was about two kinds of R2: centered, that is taught at school, and uncentered R2. It was shown, that the R statistical package switches to the latter for zero-intercept models, as this one is always positive.

Overall R2 does not have to be always lower than zero. It will be so only if the model predicts worse than just the global mean (and there you have no intercept - no global mean).

I have no rights to this picture Source: LinkedIn, posted by Adrian Olszewski, I have no rights to this picture, just saved it to my computer.

Yes, it is difficult to read, but there is a whole discussion on the LinkedIn, where mr Olszewski explains the way R handles this situation. I shared this because it gives two examples when the centered R2 takes negative value. This speaks to my mind when I see it. The points fluctuate at y=10, so the global mean is about so. Because we force the line to go through (0,0), the model performs much worse than the global mean and the R2 is -704. On the second picture the situation is less critical, and here R2 = -0.86. I know this is not an explanation but it contains two illustrations.