7
$\begingroup$

With an intercept in linear regression, I know that $R^2$ is bounded by [0,1]. Without an intercept, I know that $R^2$ can be negative. What is the lower bound for this case? And is the upper bound still 1 for this case?

$\endgroup$
6
  • 1
    $\begingroup$ The $R^2$ doesn't have any meaning when there is no intercept so it can be anything but the number itself is meaningless. There's a good paper on this. If I can find it, I'll link to it. $\endgroup$
    – mlofton
    Aug 19, 2020 at 2:19
  • $\begingroup$ I found it. web.ist.utl.pt/~ist11038/compute/errtheory/,regression/… $\endgroup$
    – mlofton
    Aug 19, 2020 at 2:20
  • $\begingroup$ You might also enjoy this paper on no-intercept regression: tandfonline.com/doi/full/10.1080/00224065.1977.11980770 $\endgroup$ Nov 12, 2023 at 6:32
  • 1
    $\begingroup$ @ShawnHemelstrand will check it out. interesting that you just commented. it's a 3+ year old post with no recent noise, and I literally referred back to this yesterday and now I see your notification $\endgroup$
    – roulette01
    Nov 13, 2023 at 2:20
  • 1
    $\begingroup$ Zhanxiong just posted an answer yesterday, which is why I saw and commented about the link. So its not quite so random after all. $\endgroup$ Nov 13, 2023 at 2:23

3 Answers 3

5
$\begingroup$

There is no lower bound.

Consider an intercept-free regression on two points, $(0,a)$ and $(1,a+2)$. By forcing the regression line to go through the origin, you can make the fit as bad as you want by making $a$ very large.

I think the upper bound remains $1$, but I confess that I’m not sure, though I find it hard to believe otherwise.

EDIT

The upper bound is $1$. Consider a regression through $(0,0)$, $(1,1)$, and $(2,2)$.

$\endgroup$
5
  • $\begingroup$ Ah so the lower bound is negative infinity basically? $\endgroup$
    – roulette01
    Aug 19, 2020 at 2:45
  • $\begingroup$ It’s fairly awkward phrasing to me to call $-\infty$ the lower bound (more interestingly, the greatest lower bound), but I think you get the idea. $\endgroup$
    – Dave
    Aug 19, 2020 at 2:47
  • $\begingroup$ What happens if the two points were both on the y axis? Like say $(0, a), (0,-a)$. It seems you cannot even get a fit for these 2 points? $\endgroup$
    – roulette01
    Aug 19, 2020 at 2:55
  • $\begingroup$ Throw that in the $(X^TX)^{-1}X^Ty$ matrix and see what happens! I’ve never considered myself his particular problem. Something is going to go wrong, but I don’t know what. $\endgroup$
    – Dave
    Aug 19, 2020 at 2:58
  • $\begingroup$ yeah the system is singular $\endgroup$
    – roulette01
    Aug 19, 2020 at 4:00
5
$\begingroup$

+1 to Dave's answer about the lower bound.

And yes, the upper bound is still $1$. In

$$ R^2 = 1-\frac{SS_\text{res}}{SS_\text{tot}}, $$

the numerator and the denominator of the fraction are both sums of squares, so they are nonnegative, so the entire fraction must be nonnegative. So the entire expression is maximized (with a value of $1$) if the fraction is zero.

And this will happen in a regression without an intercept if and only if all data points lie on a line that goes through the origin. So this upper bound will be achieved in a specific case.

$\endgroup$
1
  • $\begingroup$ Ooo right. that should've been obvious to me.. $\endgroup$
    – roulette01
    Aug 19, 2020 at 5:45
1
$\begingroup$

In order to determine the lower/upper bound of the $R^2$ for a linear model $y = X\beta + \varepsilon$ without an intercept (i.e., $X$ does not contain the column $e$ of all ones), it is helpful to first review why $R^2 \in [0, 1]$ when $X$ contains $e$.

In matrix form, regardless $X$ contains $e$ or not, $R^2$ can be written as (the first equality is by definition, the second equality is by algebra) \begin{align*} R^2 = 1 - \frac{y^\top(I - H)y}{y^\top(I - n^{-1}ee^\top)y} = \frac{y^\top(H - n^{-1}ee^\top)y}{y^\top(I - n^{-1}ee^\top)y}, \tag{$\star$}\label{0} \end{align*} where $H = X(X^\top X)^{-1}X^\top$ is hat matrix.

When $e$ is a column of $X$, one can easily verify that $He = e$ (geometrically, this is very obvious: when you project a vector onto a space that contains this vector, you get the vector itself), whence the matrix $H - n^{-1}ee^\top$ is idempotent: \begin{align*} (H - n^{-1}ee^\top)^2 = H^2 - n^{-1}Hee^\top - n^{-1}ee^\top H + n^{-1}ee^\top = H - n^{-1}ee^\top. \end{align*} Together with that $H - n^{-1}ee^\top$ is symmetric, it follows that $$y^\top(H - n^{-1}ee^\top)y = \|(H - n^{-1}ee^\top)y\|^2 \geq 0. \tag{1}\label{1} $$ This shows that $R^2 \geq 0$. On the other hand, since \begin{align*} y^\top(I - n^{-1}ee^\top)y - y^\top(H - n^{-1}ee^\top)y = y^\top(I - H)y = \|(I - H)y\|^2 \geq 0 \end{align*} and $y^\top(I - n^{-1}ee^\top)y > 0$ always hold (unless $y$ is a multiple of $e$, for which case $y$ has no variability and we usually rule this case out), we see that $R^2 \leq 1$. Note that since $I - H$ is idempotent and symmetric regardless $X$ contains $e$ or not, this part of the proof also applies to the intercept-free case. That is, $1$ continues to be the upper bound of intercept-free $R^2$.

Now when $e$ is not a column of $X$, the key property $He = e$ in general no longer holds. Consequently, $\eqref{1}$ becomes invalid (i.e., $y^\top(H - n^{-1}ee^\top)y$ can be negative). However, as argued above, $y^\top(I - n^{-1}ee^\top)y$ is always positive. As a result, the right hand side of $\eqref{0}$ may become negative.

One step back, then does it have a fixed lower bound (possibly negative)? The answer is no. For example, consider the case $n = 2, p = 1$, $X = \begin{bmatrix}0 \\ 1 \end{bmatrix}$, $y = \begin{bmatrix} y_1 \\ y_2 \end{bmatrix}$. Direct evaluation of $\eqref{0}$ yields \begin{align*} R^2 = \frac{-(y_1 + y_2)^2 + 2y_2^2}{(y_1 - y_2)^2}. \end{align*} Setting $y_2 = M$, $y_1 = M + 1$ in the above expression gives \begin{align*} R^2 = -2M^2 - 4M - 1 \end{align*} which tends to $-\infty$ as $M \to \infty$. This shows that intercept-free $R^2$ cannot be bounded from below.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.