6
$\begingroup$

Suppose we have a linear regression and we calculate $R^2 = 0.81$. What do we mean when we say "the relationship between two variables $X$ and $Y$ explains $81\text%$ of the variation of the data"? Especially this word "explains" confuses me, since the linear regression just reduces the variance in the observed data $Y$ by $81\text%$. It doesn't explain anything.

$\endgroup$
1
  • 1
    $\begingroup$ Statistics educators have recently noted that many students attempt, to their detriment, to interpret technical terms in familiar colloquial senses. (Think, for instance, of the many misinterpretations of "independent.") As an ostensible math person (judging from your user name) you probably know well that the only valid interpretation of a technical term comes through its definition and not through what the commonplace meaning of the word might be. The discipline of seeking out clear definitions will continue to serve you well in statistics. $\endgroup$
    – whuber
    Jul 20, 2022 at 22:54

3 Answers 3

5
$\begingroup$

It's an expression that is often used as short-hand or conventional jargon.

Anyone who finds it puzzling should feel that way! Some people say "accounts for" instead as a usage supposedly a little softer.

What is meant by explanation any way? This is a long-standing topic in philosophy (epistemology and philosophy of science) going back at least to Aristotle. Most disciplines have their own literature around the topic. There continue to be entire books published on that.

At a statistical end of the subject even introductory texts often emphasise that there can be nonsense correlations (e.g. when variables change over time in the same way, for quite different reasons), which alone underline that high correlations don't necessarily point to anything substantive and so aren't themselves explanatory. Or, rather there is an explanation, but it is not interesting: Ministers' salaries and the consumption of alcohol are both increasing over time, so a correlation is unsurprising, but it's still a coincidence.

Often an explanation to be satisfying would depend also on evidence on cause, process, mechanism or behaviour: choose wording to taste.

At this point it is customary to assert that "correlation doesn't prove causation", which also and fairly is often mentioned in introductory accounts. But although bang on, that put-down can be a little cheap. The truth is that proof of causation is often very difficult; meanwhile, correlations are often what we have, and what we start from. It often took centuries of hard science before sound explanations emerged for simple phenomena. Many diseases weren't explained until bacteria and viruses were identified, and many still remain mysterious at least in part. Although the behaviour was familiar to early humans, a framework for understanding the path of a dropped or thrown stone awaited Galileo and Newton.

Also, why single out correlation? The popularity of this saying depends on assonance as well as appropriateness. Generalized linear models are not causation. Support vector machines are not causation. Don't trip off the tongue so easily, do they?

To the main point: there is no implication that a formula, even one that fits the pattern of variation well, has explanatory content in any subject-matter sense. But as people say near where I live "Owt's better than nowt" (Something's better than nothing). $R^2$ with a certain number is more precise than "there is a strong relationship", just as $R^2$ with a low number and a claim of a strong relationship gives you guidance on what to think. In some areas of physics $R^2$ that isn't very high indicates incompetent experimenters; in some areas of social science a very high $R^2$ indicates faking of data or a silly question.

More subtly, the converse can be true. A formula can be derived in theory and then found to fit the data and that's first-rate science. But in some fields theory is weaker than the existence of equations may appear to imply. For example, a linear relation may be postulated on the grounds that variables are known or presumed to change together, a linear relation is the simplest we can think of, and we don't have good reason to suggest a different functional form. (This can be true in several fields, but somehow it seems common in econometrics.)

Other way round, if you say $R^2 =0.81$, then that can be quite enough for many readers who know about it. The "explains" or "accounts for" wording is for people who ask for more comment on what you're measuring, which can be reasonable and unreasonable at the same time. It's the old business of explaining in "plain English" what you''re doing, to which a riposte is that you would not need statistics if plain English would do. (Or any other language, naturally.)

As for whether it is good jargon to use, look and ask around you. Practices vary!

  • Some fields or groups just avoid "explains" altogether. Commentary might run that $R^2$ is whatever number, and encouraging, surprising, disappointing, whatever, but writers know what $R^2$ is and consider that to be covered in texts and courses, so no need to explain further.

  • Some use "explains" as a term of art, and if you object that the equation is not an explanation in any other sense, people may reply "Indeed" or "Of course not. Everyone knows that".

  • Some would regard any use of "explains" as obfuscating technical jargon manifesting lack of insight or empathy into what is going on underneath the data. This is most common in some social sciences in which there are groups of researchers whose attitude to statistical analysis varies between suspicion and hostility. If so, or if your readers feel uncomfortable or puzzled by the usage for any other reason, it is really better avoided.

There is a different debate about whether and how far $R^2$ is useful at all, which I will leave on one side.

$\endgroup$
4
  • $\begingroup$ Thank you so much for such a great answer! So it would be absolutely right to claim that your brand new model with $R^2 = 0,81$ doesn't explain anything about the observed variable $Y$, it just describes $Y$ better in the sense that the average distance of $Y$ from your model's plot is now smaller by $81\text%$. So your model just fits the data better by $81\text%$ in this sense and this is what we interpret as an "explanation", right? $\endgroup$
    – mathgeek
    Sep 28, 2021 at 16:38
  • $\begingroup$ "average distance" isn't quite right as we're measuring in terms of squared distances. We can perhaps epitomize in this way: $R^2$ is a descriptive measure; how far the relationship being described is also explanatory is in the mind of the beholder given what the variables are and what the context is. Or even shorter: "explains" is a term of art, not a philosophical claim. $\endgroup$
    – Nick Cox
    Sep 28, 2021 at 16:42
  • $\begingroup$ Yes, but "explains" here precisely means "fits better in the sense that average squared distance is less", right? That's how you interpret "explains $81\text%$ of the variation of the data". I mean, when you use the word "explain" that's what you think about, right? $\endgroup$
    – mathgeek
    Sep 28, 2021 at 16:51
  • $\begingroup$ I don't think I want to try yet another rewording. You're now focusing on what I customarily say and think, and so now the question shifts. First, in my own work I try to avoid "explains" for all the reasons stated, that it seems to imply more than is defensible and is likely to be misunderstood. Second, when I think about a relationship I try always to think about whether it makes sense (is interesting, informative, trivial, whatever) as well as how close it is. If it's other people's data and I don't know the context I encourage them similarly. $\endgroup$
    – Nick Cox
    Sep 28, 2021 at 17:05
5
$\begingroup$

Say you fit the model: $$ Y_i=\beta_0+\beta_1X_i $$ and you get: $$ R^2=0.81 $$ This means that your independent variable $X$ accounts for 81% of the variability in your dependent variable $Y$. Or, in other words, 81% of the variability in $Y$ is explained by the variability in $X$.

This is easy to see if you look at how $R^2$ is calculated: $$ R^2=1-\frac{SS_{residuals}}{SS_{total}} $$ where $SS_{total}$ is the total sum of squares, which represents the total variability of your data (in particular, the total variability of your dependent variable $Y$); and $SS_{residuals}$ is the residual sum of squares, which is a measure of the discrepancy between your data and the model. You can see how the lower $SS_{residuals}$, the better i.e. the higher $R^2$ will be.

Finally, recall that the total sum of squares can be partitioned as follows: $$ SS_{total}=SS_{regression}+SS_{residuals} $$ You can see how $SS_{residuals}$ represents the variability that's left after you remove the variability explained by the model, because $SS_{residuals}=SS_{total}-SS_{regression}$.

Hence, $\frac{SS_{residuals}}{SS_{total}}$ is the proportion of the total variability that isn't explained by the model, and therefore $1-\frac{SS_{residuals}}{SS_{total}}$ is the proportion of the total variability that is explained by the model.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for such a detailed answer! But a lot of what you said I already knew. I just asked about the meaning of this word "explain" in terms of $R^2$. And in your explanation you used the phrase "account for" which is just a synonym for "explain" and it didn't clarify much. Is it just a shorthand for saying "linear regression just reduces the variance in the observed data $Y$ by $81\text%$"? $\endgroup$
    – mathgeek
    Sep 28, 2021 at 12:33
1
$\begingroup$

$R^2$ is a comparison of the variance of the error terms of two models. One model is naïve and always makes the same guess of $\bar y$ each time; this is how we get the "total sum of squares". The other model uses some features in an attempt to have less variance in the error term; this is how we get the "residual sum of squares".

In that regard, $R^2$ is a measure of the extent to which the error term variance is reduced. Some proportion of the original variance (from the naïve model) remains, but some proportion is explained by considering features.

Let's do an example. You want to know the height of a person you select from your family. Knowing nothing about the person you select, the best guess (in terms of minimizing square loss) is the mean height of people in your family; this is the naïve model. However, you know that adults tend to be taller than children. By considering age, you reduce the variability in the heights; this is the regression model of interest. Some of the reason the raw height values vary is because the subjects have different ages; age explains some of the variability in heights.

In math, and (shamelessly) taking some of a previous post of mine...

Notation

$y_i$ is observation $i$ of some response variable $Y$.

$\hat{y}_i$ is the value of $y_i$ predicted by the regression.

$\bar{y}$ is the average of all observations of the response variable.

$$ y_i-\bar{y} = (y_i - \hat{y_i} + \hat{y_i} - \bar{y}) = (y_i - \hat{y_i}) + (\hat{y_i} - \bar{y}) $$

$$( y_i-\bar{y})^2 = \Big[ (y_i - \hat{y_i}) + (\hat{y_i} - \bar{y}) \Big]^2 = (y_i - \hat{y_i})^2 + (\hat{y_i} - \bar{y})^2 + 2(y_i - \hat{y_i})(\hat{y_i} - \bar{y}) $$

$$SSTotal := \sum_i ( y_i-\bar{y})^2 = \sum_i(y_i - \hat{y_i})^2 + \sum_i(\hat{y_i} - \bar{y})^2 + 2\sum_i\Big[ (y_i - \hat{y_i})(\hat{y_i} - \bar{y}) \Big]$$

$$ :=SSRes + SSReg + Other $$

In OLS regression, $Other = 0$.

$$R^2= 1-\dfrac{SSRes}{SSTotal}\\ = \dfrac{SSTotal -SSRes-Other}{SSTotal} \\ = \dfrac{SSReg}{SSTotal} \\ =\dfrac{SSReg/n}{SSTotal/n} \\ =\dfrac{ \mathbb Var(\epsilon_{model}) }{ \mathbb Var(\epsilon_{naïve}) }$$

The claim about "proportion of variance explained" only applies in-sample (related to the $Other$ term not being zero), so if we consider the sample to be a population, then dividing by $n$ gives the exact variance.

$\endgroup$
2
  • $\begingroup$ Why the claim about "proportion of variance explained" only applies in-sample is a topic for a separate question, one for which I would like to post an answer (and I might even post a question and a self-answer) that I would then link here. $\endgroup$
    – Dave
    Jul 20, 2022 at 21:51
  • $\begingroup$ I read this and was reminded of stats.stackexchange.com/a/551916/9330 -- ah, no wonder it felt familiar, you linked to it in your answer! $\endgroup$
    – Adrian
    Jul 20, 2022 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.