Ok, I thought I'd follow up on this. I've been struggling with the answers here a bit, and have come to some better understanding of the problem. For posterity, I also think that a full explanation of _why there are two different forms of this equation for R^2 would be beneficial to anyone who stumbles upon this thread. I don't know if this is common knowledge, or what--no one seems to explain (possibly a lot of people just don't know, or possibly it's so basic that it's expected that people 'just know') WHY there are two forms for R^2. This includes several sets of lecture notes by professors at major universities: perhaps I'm just not looking in the right places.
The reason for the two different equations above comes from the fact that you're comparing the model against the null hypothesis. The null hypothesis is "there exists zero relationship between the dependent and independent variables". This means you're taking the slope to be zero. Another way to say this is that you're comparing the regression model you build to a nested model with one fewer parameters.
Now, suppose we have a set of data with one independent variable (x) and one dependent variable (y). We have two choices:
- We choose to model the relationship between x and y with a two parameter linear model (i.e., $\hat{y}_i = a_0 + a_1 \hat{x}_i + \epsilon_i$). The null hypothesis is $\hat{y}_i = a_0 + \epsilon_i$, and $\bar{y} \neq 0$ in general. Thus the appropriate form of $R^2$ to use is:
$$
R^2 = 1- \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2}
$$
- We choose to model the relationship between x and y with a one parameter linear model, namely $\hat{y}_i = a_1 \hat{x}_i + \epsilon_i$. The null hypothesis is that there is no relationship between x and y, thus the correct null hypothesis is $\hat{y}_i = \epsilon_i$. In other words, the null hypothesis is just white noise. Clearly, $\mathbb{E}(y) = 0$, thus the correct form of $R^2$ is
$$
R^2 = 1- \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i y_i^2}
$$
A good way to think about this is the following: suppose the null hypothesis were (100%) correct, and there truly were no relationship between x and y. What would we expect? If anything's fair, the answer is "We expect $R^2=0$."
- In the case where we choose a two-parameter model, we expect that $\bar{y} = \hat{y}_i = a_0$. If this isn't obvious, try drawing the picture with the model value under the null hypothesis $\hat{y}_i$, the data point as $y_i$, and the average $\bar{y}$. If the model is correct (i.e., number of data points -> infinity), then you should be able to see graphically that $\bar{y} = \hat{y}_i = a_0$, in the case where the null hypothesis is true.
- Conversely, using the same picture as above, $\hat{y}_i = \bar{y} = 0$. There's a slight rub here, because you have to worry about how these things go to zero. L'Hopital will tell you that, in this case at least, $\lim 0/0 = 0$, and everything is ok.
You can see why you get funny things happening with $R^2$ (like negative values) if you use the wrong form of the equation. I noticed it first because the statsmodels
package in Python does one thing, and R
does something else: it pains me to say, but R
is right and statsmodels
is wrong. (Well, not really "pains"...)
I would love for some feedback on this intuition. I have only found one reference where this is explained explicitly. Please see this pdf file (download here), Section 5.3.6. Additionally, the other linked answer on stackexchange alludes to this fact, but the reasoning wasn't completely clear to me (no offense to the person who answered the question, it is a very well-written response, and I can be dense at times! ).
Again, please correct my reasoning in the comments, and I will amend the post until it is acceptable.