Is there some intuitive explanation for this terminology? Why is it this way, and not the predictor(s) being regressed on the outcome?

Ideally I'm hoping that a proper explanation of why this terminology exists will help students remember it, and stop them from saying it the wrong way around.

  • $\begingroup$ Do we? I'm not sure I've ever said that - and I've discussed regression a lot. If you know someone who does say it, maybe you could ask them. (I have on occasion said "regressed on" -- but onto would sound somewhat odd to me) $\endgroup$ – Glen_b Apr 15 '16 at 6:44
  • $\begingroup$ Thanks - I did mean "on" and not "onto". I've fixed that now. $\endgroup$ – user1205901 Apr 15 '16 at 8:00
  • $\begingroup$ Related: Why are regression problems called “regression” problems? $\endgroup$ – amoeba May 9 '16 at 10:53

I do not know what the etymology of "is regressed on" is but here is the interpretation that I have in mind when I am saying or hearing this expression. Consider the following figure from The Elements of Statistical Learning by Hastie et al.:

regression is projection

In its core, linear regression amounts to orthogonal projection of $\mathbf y$ on (onto) $\mathbf X$, where $\mathbf y$ is the $n$-dimensional vector of observations of the dependent variable and $\mathbf X$ is the subspace spanned by the predictor vectors.

This is a very useful interpretation of linear regression.

Since $y$ is being projected on $X$, that is what I think when I hear that $y$ is "regressed on" $X$. From this point of view, it would make less sense to say that $X$ is regressed on $y$ or that $y$ is regressed "against" or "with" $X$.

Ideally I'm hoping that a proper explanation of why this terminology exists will help students remember it, and stop them from saying it the wrong way around.

As I said, I doubt that this is an explanation of why this terminology exists (perhaps only of why it persists?), but I am sure it can help students remember it.

  • 2
    $\begingroup$ +1. Depends on the students! This is clearly a valid and fruitful way to talk and think at intermediate or advanced levels. Whether it's responsible for the terminology "on" I do wonder. It's not so long ago that you could find regression texts with almost no diagrams, let alone a strongly visual or geometric approach, even though that is now utterly standard, whereas I think this terminology goes back some decades. $\endgroup$ – Nick Cox Apr 15 '16 at 11:17
  • $\begingroup$ (+1) The only way that I got the concept of regression through my skull is thinking of it as the projection of $y$ onto the column space $C(A)$ of the model matrix, which I think it is the geometric interpretation you are showing. $\endgroup$ – Antoni Parellada Apr 16 '16 at 1:54
  • 1
    $\begingroup$ This is a very good statistical reason for using the terminology. The social or linguistic reasons why it's popular could be different! $\endgroup$ – Nick Cox Apr 18 '16 at 11:38
  • $\begingroup$ Just to be clear: I fully agree with what @NickCox said in the comments here. $\endgroup$ – amoeba Apr 18 '16 at 12:15

I've often used and heard this way of speaking. I'd guess that the sequence mentioning the outcome or response before the predictors follows from conventions in writing, using words or using notation or mixing the two, all the way up to

$Y = X\beta$

setting aside the equally interesting (or uninteresting!) question of what we call different kinds of variables.

But it seems equally valid mathematically and statistically to mention the predictors first, just as many mathematicians write mappings or functions with arguments first.

What often perhaps drives the sequence we use in statistical discussions is that scientifically or practically we usually have a clear idea of what we are trying to predict -- it is mortality, or income, or wheat yield, or votes in an election, or whatever -- while the pool of potential or actual predictors may not be so clear. Even if it is clear, it makes sense to mention the important things first. What are you trying to do? Predict whatever. How are you going to do it? Use some or all of these variables.

I don't have a story for "on" rather than any other word that would fit. I don't hear "regressed against" or "regressed with". There may be no logic here, just memes passed on along in textbooks, teaching and discussions.

In general, watch out. Consider a related issue, the meaning of "versus". I was brought up to say "plot $y$ [vertical axis variable] against (or versus) $x$ [horizontal axis variable]" and the reverse sounds singularly odd to me. Nevertheless people with considerable experience and expertise have it the other way round. Sometimes, this kind of difference might be traced to charismatic and idiosyncratic teachers who you have imitated ever since you sat at their feet.

  • $\begingroup$ +1. But my personal interpretation of "regressed on" is via "projected on", see my answer. I wonder if many people think about this expression this way, or is it only me. $\endgroup$ – amoeba Apr 15 '16 at 11:06

1) The term regression comes from the fact that in the usual simple linear regression model:

$y = \alpha + \beta x + \epsilon$

that unless the outcome, $y$, and predictor, $x$, variables are perfectly correlated, the fitted values, $\hat{y}$, are closer to the mean of the outcome, $\bar{y}$, (after standardization) than the predictor variable, $x$, is to its mean, $\bar{x}$ (after standardization). Thus the outcome exhibits regression toward the mean.

$|\hat{y} - \bar{y}| / s_y < |x - \bar{x}| / s_x $

For example if we use the BOD data frame built into R then:

fm <- lm(demand ~ Time, BOD)
with(BOD, all( abs(fitted(fm) - mean(demand)) / sd(demand) < abs(scale(Time))))
## [1] TRUE

For a a proof see: https://en.wikipedia.org/wiki/Regression_toward_the_mean

2) The term on comes from the fact that the fitted values are the projection of the outcome variable onto the subspace spanned by the predictor variables (including the intercept) as further explained in many sources such as http://people.eecs.ku.edu/~jhuan/EECS940_S12/slides/linearRegression.pdf .

  • $\begingroup$ I'm pretty sure that's not where the term regression comes from. In an early use of the term son's height was regressed on father's height; due to mean reversion findings showed sons of tall fathers tended to regress to the mean. $\endgroup$ – PaulB Sep 11 '17 at 19:42

Personally, when it comes to explaining terminology, I find the definition of the term itself always helps, especially when explaining to students. The actual definition of the word regress is:

"return to a former or less developed state".

So one way to explain I guess would be the following:

"Thinking of the outcome as the fully developed state, we try to explain the outcome by using less developed states, i.e. the independent variables. Thus the outcome is regressed on the predictors."

Hope that helps.

  • 1
    $\begingroup$ There is more than one "actual definition". I would suggest that in statistical science the technical definition of regression as fitting a model (by default a linear model) is now primary and the historic sense.as captured by "regress to the mean", which remains interesting and some times useful, is secondary. I don't find it helpful to think that predictors in general are "less developed states", e.g. there is no sense in which predictor rainfall is a less developed state of outcome wheat yield. Either way, I don't see how this explains the expression. $\endgroup$ – Nick Cox Apr 15 '16 at 15:55
  • $\begingroup$ I see your point completely. Is there a way you could explain regression through the definition I posted? Because the way I would think of "less developed" isn't in the sense of rainfall being less developed than wheat yield, but more as the a something that can partly explain wheat yield. $\endgroup$ – EhsanF Apr 18 '16 at 8:59
  • 1
    $\begingroup$ If "less developed" doesn't mean less developed, I can't see that the wording helps at all. $\endgroup$ – Nick Cox Apr 18 '16 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.