22
$\begingroup$

I was studying linear regression and got stuck in r-squared. I know how to calculate r-squared like a machine, but I want to understand r-squared in human language. For example, what is the meaning of r-squared = 81%? I googled and watch several tutorials and gathered some human intuition of r-squared = 81%.

r-squared = 81% means:

  • 81% less variance around the regression line than mean line
  • 81% less error between predicted values and actual values
  • Actual data is 81% close to the regression line than mean line
  • 81% better prediction of actual values using regression line than mean line

These are all human language of r-squared = 81% I got. Please correct me if I am wrong. I watched a video 1 and found another explanation of r-squared. Which is: "r-squared is the percentage of variation in 'Y' that is accounted for by its regression on 'X'"

Well, the last explanation is a bit confusing for me. Could anyone make me understand with a simple example of what this line actually means?

$\endgroup$
  • 2
    $\begingroup$ The model explains 81% of the variation in the data $\endgroup$ – Robert Long Jul 23 at 18:26
  • 3
    $\begingroup$ Related: Is $R^2$ useful or dangerous?, & especially Cosma Shalizi's notes on $R^2$ that prompted that question (pdf). $\endgroup$ – gung - Reinstate Monica Jul 23 at 18:38
  • 1
    $\begingroup$ What I often use as extended version of the comment by @RobertLong: "The response variable varies across rows. 81% of this variation is due to differences in the covariables." $\endgroup$ – Michael M Jul 23 at 19:28
43
$\begingroup$

As a matter of fact, this last explanation is the best one:

r-squared is the percentage of variation in 'Y' that is accounted for by its regression on 'X'

Yes, it is quite abstract. Let's try to understand it.

Here is some simulated data.

scatterplot

R code:

set.seed(1)
xx <- runif(100)
yy <- 1-xx^2+rnorm(length(xx),0,0.1)
plot(xx,yy,pch=19)

What we are mainly interested in is the variation in the dependent variable $y$. In a first step, let's disregard the predictor $x$. In this very simple "model", the variation in $y$ is the sum of the squared differences between the entries of $y$ and the mean of $y$, $\overline{y}$:

scatterplot with mean

abline(h=mean(yy),col="red",lwd=2)
lines(rbind(xx,xx,NA),rbind(yy,mean(yy),NA),col="gray")

This sum of squares turns out to be:

sum((yy-mean(yy))^2)
[1] 8.14846

Now, we try a slightly more sophisticated model: we regress $y$ on $x$ and check how much variation remains after that. That is, we now calculate the sums of squared differences between the $y$ and the regression line:

scatterplot regression line

plot(xx,yy,pch=19)
model <- lm(yy~xx)
abline(model,col="red",lwd=2)
lines(rbind(xx,xx,NA),rbind(yy,predict(model),NA),col="gray")

Note how the differences - the gray lines - are much smaller now than before!

And here is the sum of squared differences between the $y$ and the regression line:

sum(residuals(model)^2)
[1] 1.312477

It turns out that this is only about 16% of the sums of squared residuals we had above:

sum(residuals(model)^2)/sum((yy-mean(yy))^2)
[1] 0.1610705

Thus, our regression line model reduced the unexplained variation in the observed data $y$ by 100%-16% = 84%. And this number is precisely the $R^2$ that R will report to us:

summary(model)

Call:
lm(formula = yy ~ xx)
... snip ...    
Multiple R-squared:  0.8389,    Adjusted R-squared:  0.8373 

Now, one question you might have is why we calculate variation as a sum of squares. Wouldn't it be easier to just sum up the absolute lengths of the deviations we plot above? The reason for that lies in the fact that squares are just much easier to handle mathematically, and it turns out that if we work with squares, we can prove all kinds of helpful theorems about $R^2$ and related quantities, namely $F$ tests and ANOVA tables.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1 Would it be worthwhile editing to add a liberal sprinkling of "in the data" (e.g., "reduced unexplained variation in the data"? $\endgroup$ – Alexis Jul 23 at 18:36
  • $\begingroup$ I don't think you can conclude the model prediction is 81% better. First, we could just be picky and note this doesn't say if the model is better in an L1, L2 or whatever sense? Second, and more importantly, we could just add nonsense covariates to the model which might be insignificant but would increase the R^2. I doubt any statistician would conclude this larger model is better in any sense. $\endgroup$ – kurtosis Jul 23 at 18:37
  • 1
    $\begingroup$ @kurtosis: I nowhere said that model prediction is 81% better. By "the last statement", I did not refer to the last bullet point, but to the statement below the list. Let me edit to clarify. $\endgroup$ – Stephan Kolassa Jul 23 at 18:39
  • 2
    $\begingroup$ @Alexis: thanks, that makes sense. I sprinkled a bit. $\endgroup$ – Stephan Kolassa Jul 23 at 18:41
  • 1
    $\begingroup$ @stephan-kolassa: Ah, thank goodness! Thanks for clarifying. $\endgroup$ – kurtosis Jul 23 at 23:12
8
$\begingroup$

An R-squared is the percentage of variance explained by a model. Let's say your data has a variance of 100: that is the sum of squared errors versus the mean and divided by $N-1$ (the degrees of freedom). Then you go model the data and your model has an $R^2$ of 81%. That means that the model predictions have a variance of 81. The remaining variance, 19, is the variance of your data versus the conditional mean (i.e. the variance about the regression line). Thus your first statement is correct: there is "81% less variance around the regression line than mean line."

Your second and third statements are not correct since "less error" and "closer" could easily be interpreted as using the distance between points and the regression line (and so minimizing the $L_1$ norm=absolute value of errors).

Your fourth statement is very difficult to interpret, so I do not know if you could even say it is wrong. Prediction being "81% better" is totally unclear in what it means. You have already mentioned "closer" yet we cannot conclude that this model is 81% closer to the observed data (as per the above critiques of statements #2 and #3.) Worse: we could just add many noise variables to the model. Those are likely to be insignificant yet you could find some set of them which would increase the $R^2$. (You have then overfit your data.) I doubt you or any statistician would conclude this larger model with is better in any sense. Furthermore, your model may only be trying to explain and not predict, so I would especially avoid statements like #4.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for pointing out my mistake. I edited and put 'd':) $\endgroup$ – zipper block Jul 23 at 18:42
  • 1
    $\begingroup$ Edited my answer accordingly! :-) Thanks also to @bilibraker for LaTeX-ing my answer appropriately (after I forgot to do so). $\endgroup$ – kurtosis Jul 24 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.