28
$\begingroup$

Is there some intuitive explanation for this terminology? Why is it this way, and not the predictor(s) being regressed on the outcome?

Ideally I'm hoping that a proper explanation of why this terminology exists will help students remember it, and stop them from saying it the wrong way around.

$\endgroup$
3
  • 2
    $\begingroup$ Do we? I'm not sure I've ever said that - and I've discussed regression a lot. If you know someone who does say it, maybe you could ask them. (I have on occasion said "regressed on" -- but onto would sound somewhat odd to me) $\endgroup$
    – Glen_b
    Apr 15, 2016 at 6:44
  • $\begingroup$ Thanks - I did mean "on" and not "onto". I've fixed that now. $\endgroup$ Apr 15, 2016 at 8:00
  • $\begingroup$ Related: Why are regression problems called “regression” problems? $\endgroup$
    – amoeba
    May 9, 2016 at 10:53

5 Answers 5

34
$\begingroup$

I do not know what the etymology of "is regressed on" is but here is the interpretation that I have in mind when I am saying or hearing this expression. Consider the following figure from The Elements of Statistical Learning by Hastie et al.:

regression is projection

In its core, linear regression amounts to orthogonal projection of $\mathbf y$ on (onto) $\mathbf X$, where $\mathbf y$ is the $n$-dimensional vector of observations of the dependent variable and $\mathbf X$ is the subspace spanned by the predictor vectors.

This is a very useful interpretation of linear regression.

Since $y$ is being projected on $X$, that is what I think when I hear that $y$ is "regressed on" $X$. From this point of view, it would make less sense to say that $X$ is regressed on $y$ or that $y$ is regressed "against" or "with" $X$.

Ideally I'm hoping that a proper explanation of why this terminology exists will help students remember it, and stop them from saying it the wrong way around.

As I said, I doubt that this is an explanation of why this terminology exists (perhaps only of why it persists?), but I am sure it can help students remember it.

$\endgroup$
4
  • 3
    $\begingroup$ +1. Depends on the students! This is clearly a valid and fruitful way to talk and think at intermediate or advanced levels. Whether it's responsible for the terminology "on" I do wonder. It's not so long ago that you could find regression texts with almost no diagrams, let alone a strongly visual or geometric approach, even though that is now utterly standard, whereas I think this terminology goes back some decades. $\endgroup$
    – Nick Cox
    Apr 15, 2016 at 11:17
  • 1
    $\begingroup$ (+1) The only way that I got the concept of regression through my skull is thinking of it as the projection of $y$ onto the column space $C(A)$ of the model matrix, which I think it is the geometric interpretation you are showing. $\endgroup$ Apr 16, 2016 at 1:54
  • 2
    $\begingroup$ This is a very good statistical reason for using the terminology. The social or linguistic reasons why it's popular could be different! $\endgroup$
    – Nick Cox
    Apr 18, 2016 at 11:38
  • $\begingroup$ Just to be clear: I fully agree with what @NickCox said in the comments here. $\endgroup$
    – amoeba
    Apr 18, 2016 at 12:15
8
$\begingroup$

1) The term regression comes from the fact that in the usual simple linear regression model:

$y = \alpha + \beta x + \epsilon$

that unless the outcome, $y$, and predictor, $x$, variables are perfectly correlated, the fitted values, $\hat{y}$, are closer to the mean of the outcome, $\bar{y}$, (after standardization) than the predictor variable, $x$, is to its mean, $\bar{x}$ (after standardization). Thus the outcome exhibits regression toward the mean.

$|\hat{y} - \bar{y}| / s_y < |x - \bar{x}| / s_x $

For example if we use the BOD data frame built into R then:

fm <- lm(demand ~ Time, BOD)
with(BOD, all( abs(fitted(fm) - mean(demand)) / sd(demand) < abs(scale(Time))))
## [1] TRUE

For a a proof see: https://en.wikipedia.org/wiki/Regression_toward_the_mean

2) The term on comes from the fact that the fitted values are the projection of the outcome variable onto the subspace spanned by the predictor variables (including the intercept) as further explained in many sources such as http://people.eecs.ku.edu/~jhuan/EECS940_S12/slides/linearRegression.pdf .

Note

Regarding the comment below, what the commenter is stating is what the answer already states above in formula form except that the answer states it correctly. In fact, due to the equality:

$(\hat{y} - \bar{y}) = \hat{\beta} (x - \bar{x}) $

the dependent variable is not necessarily on average closer to its mean than the predictor is to its mean unless $| \beta | < 1$ . What is true is that the dependent variable is on average fewer standard deviations from its mean than the predictor is to its as stated in the formula in the answer.

Using Galton's data to which the comment refers (which is available in the UsingR package in R) we run the regression and in fact the slope is 0.646 so the average child was closer to its mean than its parent was to its mean but that is not the general case.

library(UsingR)
fm2 <- lm(child ~ parent, galton)
coef(fm2)[[2]] # slope
## [1] 0.646

The BOD example in (1) above is one where the dependent variable is not closer to its mean unless one measures closeness in standard deviations as the slope > 1.

coef(fm)[[2]] # slope
## [1] 1.7214 

with(BOD, all( abs(fitted(fm) - mean(demand)) < abs(Time - mean(Time))))
## [1] FALSE
$\endgroup$
2
  • 1
    $\begingroup$ I'm pretty sure that's not where the term regression comes from. In an early use of the term son's height was regressed on father's height; due to mean reversion findings showed sons of tall fathers tended to regress to the mean. $\endgroup$
    – PaulB
    Sep 11, 2017 at 19:42
  • $\begingroup$ While that was true for that particular dataset that is not in general true unless you measure closeness in terms of standard deviations but that is precisely what the inequality in the answer does so perhaps you just did not recognize it. In fact the modern notion is based on the correct formulation which I have stated and not on the incorrect formulation not involving standard deviation. I have expanded on this in the Note which I have added to the end of the answer. $\endgroup$ Jan 11, 2020 at 2:33
7
$\begingroup$

I've often used and heard this way of speaking. I'd guess that the sequence mentioning the outcome or response before the predictors follows from conventions in writing, using words or using notation or mixing the two, all the way up to

$Y = X\beta$

setting aside the equally interesting (or uninteresting!) question of what we call different kinds of variables.

But it seems equally valid mathematically and statistically to mention the predictors first, just as many mathematicians write mappings or functions with arguments first.

What often perhaps drives the sequence we use in statistical discussions is that scientifically or practically we usually have a clear idea of what we are trying to predict -- it is mortality, or income, or wheat yield, or votes in an election, or whatever -- while the pool of potential or actual predictors may not be so clear. Even if it is clear, it makes sense to mention the important things first. What are you trying to do? Predict whatever. How are you going to do it? Use some or all of these variables.

I don't have a story for "on" rather than any other word that would fit. I don't hear "regressed against" or "regressed with". There may be no logic here, just memes passed on along in textbooks, teaching and discussions.

In general, watch out. Consider a related issue, the meaning of "versus". I was brought up to say "plot $y$ [vertical axis variable] against (or versus) $x$ [horizontal axis variable]" and the reverse sounds singularly odd to me. Nevertheless people with considerable experience and expertise have it the other way round. Sometimes, this kind of difference might be traced to charismatic and idiosyncratic teachers who you have imitated ever since you sat at their feet.

$\endgroup$
1
  • $\begingroup$ +1. But my personal interpretation of "regressed on" is via "projected on", see my answer. I wonder if many people think about this expression this way, or is it only me. $\endgroup$
    – amoeba
    Apr 15, 2016 at 11:06
3
$\begingroup$

As the target predicted outcome y depends on the predictor x, you can say that "is regressed on" means "is dependent on".

The word "regressed" is used instead of "dependent" because we want to emphasise that we are using a regression technique to represent this dependency between x and y.

So, this sentence "y is regressed on x" is the short format of:
Every predicted y shall "be dependent on" a value of x through a regression technique.

$\endgroup$
1
  • $\begingroup$ However, "is dependent on" is something we consider true or possibly true of a data generation process while "regressed on" is an action by statistics users. Although I answered too I am not clear that there really is a deeper explanation than that "on" is just a word often used, which is no explanation at all. $\endgroup$
    – Nick Cox
    Oct 18, 2020 at 7:46
2
$\begingroup$

Personally, when it comes to explaining terminology, I find the definition of the term itself always helps, especially when explaining to students. The actual definition of the word regress is:

"return to a former or less developed state".

So one way to explain I guess would be the following:

"Thinking of the outcome as the fully developed state, we try to explain the outcome by using less developed states, i.e. the independent variables. Thus the outcome is regressed on the predictors."

Hope that helps.

$\endgroup$
3
  • 1
    $\begingroup$ There is more than one "actual definition". I would suggest that in statistical science the technical definition of regression as fitting a model (by default a linear model) is now primary and the historic sense.as captured by "regress to the mean", which remains interesting and some times useful, is secondary. I don't find it helpful to think that predictors in general are "less developed states", e.g. there is no sense in which predictor rainfall is a less developed state of outcome wheat yield. Either way, I don't see how this explains the expression. $\endgroup$
    – Nick Cox
    Apr 15, 2016 at 15:55
  • $\begingroup$ I see your point completely. Is there a way you could explain regression through the definition I posted? Because the way I would think of "less developed" isn't in the sense of rainfall being less developed than wheat yield, but more as the a something that can partly explain wheat yield. $\endgroup$
    – EhsanF
    Apr 18, 2016 at 8:59
  • 1
    $\begingroup$ If "less developed" doesn't mean less developed, I can't see that the wording helps at all. $\endgroup$
    – Nick Cox
    Apr 18, 2016 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.