Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear model.

To paraphrase Shalizi's example: suppose you have a model $Y = aX + \epsilon$, where $a$ is known. Then $\newcommand{\Var}{\mathrm{Var}}\Var[Y] = a^2 \Var[x] + \Var[\epsilon]$ and the amount of explained variance is $a^2 \Var[X]$, so $R^2 = \frac{a^2 \Var[x]}{a^2 \Var[X] + \Var[\epsilon]}$. This goes to 0 as $\Var[X] \rightarrow 0$ and to 1 as $\Var[X] \rightarrow \infty$.

Conversely, you can get high $R^2$ even when your model is noticeably non-linear. (Anyone have a good example offhand?)

So when is $R^2$ a useful statistic, and when should it be ignored?

share|improve this question
3  
Please note the related comment thread in another recent question –  whuber Jul 20 '11 at 20:47
15  
I have nothing statistical to add to the excellent answers given (esp. the one by @whuber) but I think the right answer is "R-squared: Useful and dangerous". Like pretty much any statistic. –  Peter Flom Jul 21 '11 at 10:47
14  
The answer to this question is: "Yes" –  Fomite Apr 23 '12 at 20:52

6 Answers 6

up vote 131 down vote accepted

To address the first question, consider the model

$$Y = X + \sin(X) + \varepsilon$$

with iid $\varepsilon$ of mean zero and finite variance. As the range of $X$ (thought of as fixed or random) increases, $R^2$ goes to 1. Nevertheless, if the variance of $\varepsilon$ is small (around 1 or less), the data are "noticeably non-linear." In the plots, $var(\varepsilon)=1$.

Short range of X

Wider range of X

Incidentally, an easy way to get a small $R^2$ is to slice the independent variables into narrow ranges. The regression (using exactly the same model) within each range will have a low $R^2$ even when the full regression based on all the data has a high $R^2$. Contemplating this situation is an informative exercise and good preparation for the second question.

Both the following plots use the same data. The $R^2$ for the full regression is 0.86. The $R^2$ for the slices (of width 1/2 from -5/2 to 5/2) are .16, .18, .07, .14, .08, .17, .20, .12, .01, .00, reading left to right. If anything, the fits get better in the sliced situation because the 10 separate lines can more closely conform to the data within their narrow ranges. Although the $R^2$ for all the slices are far below the full $R^2$, neither the strength of the relationship, the linearity, nor indeed any aspect of the data (except the range of $X$ used for the regression) has changed.

Point cloud with full regression

Sliced point cloud with 10 regressions

(One might object that this slicing procedure changes the distribution of $X$. That is true, but it nevertheless corresponds with the most common use of $R^2$ in fixed-effects modeling and reveals the degree to which $R^2$ is telling us about the variance of $X$ in the random-effects situation. In particular, when $X$ is constrained to vary within a smaller interval of its natural range, $R^2$ will usually drop.)

The basic problem with $R^2$ is that it depends on too many things (even when adjusted in multiple regression), but most especially on the variance of the independent variables and the variance of the residuals. Normally it tells us nothing about "linearity" or "strength of relationship" or even "goodness of fit" for comparing a sequence of models.

Most of the time you can find a better statistic than $R^2$. For model selection you can look to AIC and BIC; for expressing the adequacy of a model, look at the variance of the residuals.

This brings us finally to the second question. One situation in which $R^2$ might have some use is when the independent variables are set to standard values, essentially controlling for the effect of their variance. Then $1 - R^2$ is really a proxy for the variance of the residuals, suitably standardized.

share|improve this answer
13  
What an amazingly thorough and responsive answer by @whuber –  Peter Flom Jul 21 '11 at 10:40
    
Doesn't AIC and BIC explicitly adjust for the number of estimated parameters? If so, doing a comparison to and unadjusted R^2 seems unfair. So I ask, does your critique hold adjusted R^2? It seems like if you were penalized for 'slicing' that adjusted R^2 would be able to go back to telling you about the goodness of fit of the model. –  rpierce Jul 22 '11 at 13:56
3  
@dr My critique applies perfectly to adjusted $R^2$. The only cases where there's much of a difference between $R^2$ and the adjusted $R^2$ are when you are using loads of parameters compared to the data. In the slicing example there were almost 1,000 data points and the slicing added only 18 parameters; the adjustments to $R^2$ wouldn't even affect the second decimal place, except possibly in the end segments where there were only a few dozen data points: and it would lower them, actually strengthening the argument. –  whuber Jul 22 '11 at 14:01
2  
The answer to the question in your first comment ought to depend on your objective and there are several ways to interpret "testing for a linear relationship." One is, you want to test whether the coefficient is nonzero. Another is, you want to know whether there is evidence of nonlinearity. $R^2$ (by itself) isn't terribly useful for either, although we know that a high $R^2$ with plenty of data means their scatterplot looks roughly linear--like my second one or like @macro's example. For each objective there is an appropriate test and its associated p-value. –  whuber Jul 22 '11 at 16:41
2  
For your second question we ought to wonder what might be meant by "best" linear fit. One candidate would be any fit that minimizes the residual sum of squares. You could safely use $R^2$ as a proxy for this, but why not examine the (adjusted) root mean square error itself? It's a more useful statistic. –  whuber Jul 22 '11 at 16:44

Your example only applies when the variable $\newcommand{\Var}{\mathrm{Var}}X$ should be in the model. It certainly doesn't apply when one uses the usual least squares estimates. To see this, note that if we estimate $a$ by least squares in your example, we get:

$$\hat{a}=\frac{\frac{1}{N}\sum_{i=1}^{N}X_{i}Y_{i}}{\frac{1}{N}\sum_{i=1}^{N}X_{i}^{2}}=\frac{\frac{1}{N}\sum_{i=1}^{N}X_{i}Y_{i}}{s_{X}^{2}+\overline{X}^{2}}$$ Where $s_{X}^2=\frac{1}{N}\sum_{i=1}^{N}(X_{i}-\overline{X})^{2}$ is the (sample) variance of $X$ and $\overline{X}=\frac{1}{N}\sum_{i=1}^{N}X_{i}$ is the (sample) mean of $X$

$$\hat{a}^{2}\Var[X]=\hat{a}^{2}s_{X}^{2}=\frac{\left(\frac{1}{N}\sum_{i=1}^{N}X_{i}Y_{i}\right)^2}{s_{X}^2}\left(\frac{s_{X}^{2}}{s_{X}^{2}+\overline{X}^{2}}\right)^2$$

Now the second term is always less than $1$ (equal to $1$ in the limit) so we get an upper bound for the contribution to $R^2$ from the variable $X$:

$$\hat{a}^{2}\Var[X]\leq \frac{\left(\frac{1}{N}\sum_{i=1}^{N}X_{i}Y_{i}\right)^2}{s_{X}^2}$$

And so unless $\left(\frac{1}{N}\sum_{i=1}^{N}X_{i}Y_{i}\right)^2\to\infty$ as well, we will actually see $R^2\to 0$ as $s_{X}^{2}\to\infty$ (because the numerator goes to zero, but denominator goes into $\Var[\epsilon]>0$). Additionally, we may get $R^2$ converging to something in between $0$ and $1$ depending on how quickly the two terms diverge. Now the above term will generally diverge faster than $s_{X}^2$ if $X$ should be in the model, and slower if $X$ shouldn't be in the model. In both case $R^2$ goes in the right directions.

And also note that for any finite data set (i.e. a real one) we can never have $R^2=1$ unless all the errors are exactly zero. This basically indicates that $R^2$ is a relative measure, rather than an absolute one. For unless $R^2$ is actually equal to $1$, we can always find a better fitting model. This is probably the "dangerous" aspect of $R^2$ in that because it is scaled to be between $0$ and $1$ it seems like we can interpet it in an absolute sense.

It is probably more useful to look at how quickly $R^2$ drops as you add variables into the model. And last, but not least, it should never be ignored in variable selection, as $R^2$ is effectively a sufficient statistic for variable selection - it contains all the information on variable selection that is in the data. The only thing that is needed is to choose the drop in $R^2$ which corresponds to "fitting the errors" - which usually depends on the sample size and the number of variables.

share|improve this answer
2  
+1 Lots of nice points. The calculations add quantitative insights to the previous replies. –  whuber Aug 23 '11 at 16:44

If I can add an example of when $R^2$ is dangerous. Many years ago I was working on some biometric data and being young and foolish I was delighted when I found some statistically significant $R^2$ values for my fancy regressions which I had constructed using stepwise functions. It was only afterwards looking back after my presentation to a large international audience did I realize that given the massive variance of the data - combined with the possible poor representation of the sample with respect to the population, an $R^2$ of 0.02 was utterly meaningless even if it was "statistically significant"...

Those working with statistics need to understand the data!

share|improve this answer
4  
+1 Very interesting example (and a good story). –  whuber Jan 31 '12 at 16:58
8  
No statistic is dangerous if you understand what it means. Sean's example has nothing special to do with R square it is the general problem of being enamored with statistical significance. When we do statistical testing in practice we are only interested in meaningful differences. Two populations never have identical distributions. If they are close to equal we don't care. With very large sample sizes we can detect small unimportant differences. That is why in my medical research consulting I emphasize the difference between clinical and statistical significance. –  Michael Chernick May 6 '12 at 1:16
7  
Initially my clients often thin that statistical significance is the goal of the research. They need to be shown that it is not the case. –  Michael Chernick May 6 '12 at 1:16

When you have a single predictor $R^{2}$ is exactly interpreted as the proportion of variation in $Y$ that can be explained by the linear relationship with $X$. This interpretation must be kept in mind when looking at the value of $R^2$.

You can get a large $R^2$ from a non-linear relationship only when the relationship is close to linear. For example, suppose $Y = e^{X} + \varepsilon$ where $X \sim {\rm Uniform}(2,3)$ and $\varepsilon \sim N(0,1)$. If you do the calculation of

$$ R^{2} = {\rm cor}(X, e^{X} + \varepsilon)^{2} $$

you will find it to be around $.914$ (I only approximated this by simulation) despite that the relationship is clearly not linear. The reason is that $e^{X}$ looks an awful lot like a linear function over the interval $(2,3)$.

share|improve this answer
    
To the remarks below by Erik and Macro I don't think anyone has it out for me and it is probably better to have one combined answer instead of three separate ones but why does it matter to the point that so much discussion centers around how you write things and where you write it instead of fcusing on what is said? –  Michael Chernick May 6 '12 at 4:13
7  
@MichaelChernick, I don't think there is "so much" discussion about how one writes things. The guidelines we've tried to help you with are more along the lines of "if everyone did that, this site would be very disorganized and hard to follow". It may seem like there is a lot of discussion about these things, but that's probably just because you've been a very active participant since you joined, which is great, since you clearly bring a lot to the table. If you want to talk more about this, consider starting a thread on meta rather than a comment discussion under my unrelated answer :) –  Macro May 6 '12 at 4:40

One situation you would want to avoid $R^2$ is multiple regression, where adding irrelevant predictor variables to the model can in some cases increase $R^2$. This can be addressed by using the adjusted $R^2$ value instead, calculated as

$\bar{R}^2 = 1 - (1-R^2)\frac{n-1}{n-p-1}$ where $n$ is the number of data samples, and $p$ is the number of regressors not counting the constant term.

share|improve this answer
11  
Note that adding irrelevant variables is guaranteed to increase $R^2$ (not just in "some cases") unless those variables are completely collinear with the existing variables. –  whuber Jan 9 '12 at 18:57
  1. A good example for high R square with a nonlinear is the quadratic function y=x^2 restricted to the interval [0,1]. With 0 noise it will not have an R square of 1 if you have 3 or more points since they will not fit perfectly on a straight line. But if the design points are scattered uniformly on the [0, 1] the R square you get will be high perhaps surprisingly so. This may not be the case if you have a lot of points near 0 and alot near 1 with little or nothing in the middle.

  2. R square will be poor in the perfect linear case if the noise term has a large variance. So you can take the model Y= x + e which is technically a perfect linear model but let the variance in e tend to infinity and you will have R square going to 0. Inspite of its deficiencies R square does measure the percentage of variance explained by the data and so it does measure goodness of fit. A high R square means a good fit but we still have to be careful about the good fit being casued by too many parameters for the size of the data set that we have.

  3. In the multiple regression situation there is the overfitting problem. Add variables and R square will always increase. The adjusted R square remedies this somewhat as it takes account of the number of parameters.

share|improve this answer
6  
Michael, I've merged all your replies into a single response. Unless response are really different, posting several replies is not encouraged, as per our FAQ. –  chl May 4 '12 at 22:35
    
Someone undeleted my answers. Answers 2 and 3 were merged by chi and therefore he deleted my separate answers. It is a procedural issue not one about content. The content was preserved. I don't object but I think it would be more appropriate to let me decide whether or not to edit it –  Michael Chernick May 5 '12 at 17:35
    
Your two other replies are still deleted, AFAICT. I thought it would be helpful for you if I merged your separate replies into the present one. My comment above is just to let you know about site policy. (I'm sure you will get used to this site shortly.) Of course, you are free to edit the present material as you like now. –  chl May 5 '12 at 18:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.