8
$\begingroup$

I was asked an $R^2$ question during an interview, and I felt like I was right then, and still feel like I'm right now. Essentially the interviewer asked me if it is possible for $R^2$ to be negative for linear regression.

I said that if you're using OLS, then it is not possible because the formal definition of $R^2$ is

$$ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} $$

where $SS_{tot} = \sum_i^n (y_i - \bar{y})^2$ and $SS_{res} = \sum_i^n (y_i - \hat{y_i})^2$.

In order for $R^2$ to be negative, the second term must be greater than 1. This would imply that $SS_{res} > SS_{tot}$, which would imply that the predictive model fits worse than if you fit a straight line through the mean of the observed $y$.

I told the interviewer that it is not possible for $R^2$ to be 1 because if the horizontal line is indeed the line of best fit, then OLS fill produce that line unless we're dealing with an ill-conditioned or singular system.

He claimed that this isn't correct and that $R^2$ can still be negative, and that I could "see it easily in the case where there is no intercept." (note that all of the discussion so far was about the case WITH an intercept, which I confirmed at the beginning by asking if there are any constraints about the best line passing through the origin, which he stated "no")

I can't see this at all. I stood by my answer, and then mentioned that maybe if you used some other linear regression method, perhaps you can get a negative $R^2$.

Is there any way for $R^2$ to be negative using OLS with or without intercept? Edit: I do understand that you can get a negative $R^2$ in the case without an intercept.

$\endgroup$
8
  • $\begingroup$ stats.stackexchange.com/questions/12900/… $\endgroup$
    – dwolfeu
    Aug 7 '20 at 5:38
  • $\begingroup$ @dwolfeu Yeah I saw that post, but it doesn't necessarily answer my specific questions here. $\endgroup$
    – 24n8
    Aug 7 '20 at 5:39
  • 1
    $\begingroup$ The $R^2$ is nowadays often seen as ratio of model variance and data variance. But it is actually a correlation between the model and data. In that view the $R^2$ is necessarily between 0 and 1. As a ratio of variances, and only as 1 minus the ratio of error:data, then you can easily get negative values due to all sorts of things that might get the model to be worse than just fitting with the mean (e.g. when applying shrinking then you might include an intercept but still get worse than the mean). $\endgroup$ Aug 7 '20 at 7:24
  • 1
    $\begingroup$ What sort of job has this as an interview question? $\endgroup$ Aug 7 '20 at 7:39
  • 1
    $\begingroup$ Sorry to say, your interviewer is a moron - while $R^2$ can be negative if you force the intercept to $0$, there is never any reason to do so. If you only care about of out sample predictions, it will almost always be worse than including an intercept. If you care about parameter inference, it will always you lead to biased and inconsistent estimate for both model parameters (i.e. the estimated $\beta$) and model variance. If your explanation of the situation is true, then you very likely dodged a bullet by not getting hired. $\endgroup$
    – Repmat
    Aug 7 '20 at 8:01
7
$\begingroup$

The interviewer is right. Sorry.

set.seed(2020)
x <- seq(0, 1, 0.001)
err <- rnorm(length(x))
y <- 99 - 30*x + err
L <- lm(y~0+x) # "0" forces the intercept to be zero
plot(x, y, ylim=c(0, max(y)))
abline(a=0, b= summary(L)$coef[1], col='red')
abline(h=mean(y), col='black')
SSRes <- sum(resid(L)^2)
SSTot <- sum((y - mean(y))^2)
R2 <- 1 - SSRes/SSTot
R2 

I get $R^2 = -31.22529$. This makes sense when you look at the plot the code produces.

enter image description here

The red line is the regression line. The black line is the "naive" line where you always guess the mean of $y$, regardless of the $x$.

The $R^2<0$ makes sense when you consider what $R^2$ does. $R^2$ measures how much better the regression model is at guessing the conditional mean than always guessing the pooled mean. Looking at the graph you're better off guessing the mean of the pooled values of $y$ than you are using the regression line.

EDIT

There is an argument to be made that the "SSTot" to which you should compare an intercept-free model is just the sum of squares of $y$ (so $\sum (y_i-0)^2$), not $\sum (y_i - \bar{y})^2$. However, $R^2_{ish} = 1- \frac{\sum(y_i - \hat{y}_i)^2}{\sum y_i^2}$ is quite different from the usual $R^2$ and (I think) loses the usual connection to amount of variance explained. If this $R^2_{ish}$ is used, however, when the intercept is excluded, $R^2_{ish} \ge 0$.

$\endgroup$
11
  • $\begingroup$ Sorry, I mispoke in my original post. So I do see why the case without the intercept would have a negative $R^2$. But he insisted that this could also occur for the case WITH an intercept, and then proceeded to tell me that I could "see this by considering the case without an intercept," which doesn't make sense to me because the case without an intercept is an entirely different regression model, and I don't see how looking at that case would show me how $R^2$ can be $<0$ for the case WITH an intercept. $\endgroup$
    – 24n8
    Aug 7 '20 at 4:04
  • $\begingroup$ Just edited the OP to make this part clearer. $\endgroup$
    – 24n8
    Aug 7 '20 at 4:06
  • 3
    $\begingroup$ Perhaps he meant some kind of regularized regression or out-of-sample $R^2$. $\endgroup$
    – Dave
    Aug 7 '20 at 4:06
  • $\begingroup$ I'm curious about your plot. In the case where the intercept IS included, I know that the regression line passes through $\bar{x}, \bar{y}$, i.e., $\hat{y} = \bar{y} = \hat{\beta}_1 \bar{x} + \hat{\beta}_0$. For your case without the intercept it seems that it doesn't. I can't really read $R$, but I assume the mean of your observed values have $\bar{x} = 0.5$? $\endgroup$
    – 24n8
    Aug 7 '20 at 4:10
  • 3
    $\begingroup$ Pretty much everything you know about regression (me too) is about regression with an intercept. The sentiment on here is that the proper time to exclude the intercept, though, is pretty much never. $\endgroup$
    – Dave
    Aug 7 '20 at 4:27
4
$\begingroup$

It looks like your interview was correct.

In the case that you include an intercept it is not possible.

The easiest way to see this is to take the projection view of linear regression.

$\hat{y} = X\hat{\beta} = X(X^TX)^{-1}X^TY = P_XY$

Where $P_X$ is a orthogonal projection matrix. It projects vectors into the subspace spanned by linear combinations of $X$. You can think of this as shining a light on the vector into the linear subspace spanned by X. It maps $Y$ to the closest possible part of the subspace.

We can also define the projection onto a subspace spanned by an intercept, denoted $P_\iota$, where $\iota$ is a vector of ones.

It turns out that $P_\iota Y = \bar{y}$, a $n \times 1$ vector with the mean as each value. In other words, the best possible linear approximation to $Y$ using only combinations of constants would be the mean. That makes sense and you may have seen related results in a stats class before.

If $X$ includes an intercept then the linear subspace spanned by $X$ is a superset of the linear subspace spanned by an intercept. What this means is that since $P_X$ finds the closest approximation in the subspace and it contains the intercept subspace, then it has to be at least as close to $Y$ as the best approximation in the span of $\iota$. In other words $|Y - \hat{y}| = |Y - P_XY| \leq |Y - P_\iota Y| = |Y - \bar{y}|$ if $X$ contains the intercept (and thus the squares must also follow this inequality).

Now if we do not include an intercept, this is no longer true, because the linear span of $X$ is no longer a superset of the intercept linear space. It is thus no longer guaranteed that our prediction is at least as good as the mean.

Consider the example where $X$ is a single variable with mean 0, finite variance and is independent of $Y$, and $Y$ has some arbitrary mean $E[Y] \neq 0$ (but exists).

$\hat{\beta} = (X^TX)^{-1}X^TY \overset{p}{\to} \frac{ E[XY] }{ E[X^2] } = \frac{E[X]E[Y]}{E[X^2]} = 0$

As n gets large, the coefficient becomes arbitrarily close to zero. This means that $\hat{y} \overset{p}{\to} 0$

Using the centered $\mathcal{R}^2$ formula we get

\begin{align} 1 - \frac{\sum_{i=1}^n (y_i - \hat{y})^2}{\sum_{i=1}^n(y_i -\bar{y})^2} &= 1 - \frac{\sum_{i=1}^n (y_i - o_p(1))^2}{\sum_{i=1}^n(y_i -\bar{y})^2}\\ &\overset{p}{\to} 1 - \frac{E[Y^2]}{var(Y)}\\ & = 1 - \frac{E[Y^2]}{E[Y^2] - (E[Y])^2} \leq 0 \end{align}

So if $X$ doesn't really explain anything in $Y$, and the mean of $Y$ is far from 0, we can have a really negative $\mathcal{R}^2$

Below is some R code to simulate such a case

set.seed(2020)
n <- 10000  
y <- rnorm(n,50,1)  
x <- rnorm(n)  

mod <- lm(y ~   -1 + x)  
yhat <- predict(mod)  

R2  <- 1 - sum((y - yhat)^2)/sum((y - mean(y))^2)  
R2

$\mathcal{R^2} = -2514.479$

Edit: I agree with Dave that when we don't include an intercept it would be reasonable to argue that the uncentered $\mathcal{R}^2$ is the more natural $\mathcal{R}^2$ measure. The problem with the uncentered version is that it is not invariant to changes in the mean of the regressand (see Davidson and Mackinnon: Econometric Theory and Methods chapter 3 for discussion).

$\endgroup$
6
  • 1
    $\begingroup$ I mispoke in the OP (see edits), but the interview essentially claimed that $R^2$ can be negative in the case with an intercept. Btw, I like projection and column space perspective. It's relatively more aligned with my background (more numerical linear algebra than stats). $\endgroup$
    – 24n8
    Aug 7 '20 at 4:25
  • $\begingroup$ Yeah in that case, I believe his is wrong. $\endgroup$ Aug 7 '20 at 4:32
  • $\begingroup$ since in that case we can write $1 - \frac{SS_{res}}{SS_{tot}} = 1 - \frac{|(I_n - P_X)Y|^2}{|(I_n - P_{\iota})Y|^2}$, which as we showed the numerator must be at least as small as the denominator in the fraction. Big bummer you had to deal with that. And yes, big fan of the geometric vector space perspective. If this is more your style than traditional stats, you might enjoy more general semi-parametric methods which can be put into hilbert spaces with this kind of flavour $\endgroup$ Aug 7 '20 at 4:38
  • $\begingroup$ Btw, from a linear algebra perspective, do you know why for the case without an intercept, the regression line isn't guaranteed to pass through $(\bar{x} , \bar{y})$? Is it simply because the column space of $X$ is no longer spanned by $1 \in \mathbb{R}^n$? So $\hat{y}$ can't be written as a linear combination of $\bar{X}_{i\in \{1, \ldots, p\}}$? It seems this is along the lines, but I don't think this is sufficient. $\endgroup$
    – 24n8
    Aug 7 '20 at 4:52
  • $\begingroup$ I don't have a proper lin alg intuition here, but this is what I can say and maybe it clicks for you. $\endgroup$ Aug 7 '20 at 5:15
4
$\begingroup$

Using OLS with intercept, the only situation with negative R-squared is the following:

  1. You fit your model on a training set.

  2. You apply the model on a fresh test set, calculate the out-of-sample residuals and from there, derive the out-of-sample R-squared. The latter can be negative.

Here the dummy example in R

n <- 100
df <- data.frame(x=rnorm(n), y=rnorm(n))
train <- df[1:70, ]
test <- df[71:n, ]

# Train on train
fit <- lm(y~x, train)
summary(fit) # Multiple R-squared:  3.832e-06

# Evaluate on test
oos_residuals <- test[, "y"] - predict(fit, test)

oos_residual_ss <- sum(oos_residuals^2)
oos_total_ss <- sum((test[, "y"] - mean(train[, "y"]))^2)

1 - oos_residual_ss / oos_total_ss # -0.001413857
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.