8
$\begingroup$

Consider the function

$$ r(x) = \mathbb{E}(Y \mid X = x) $$

This has been called the regression function in a textbook I'm using. I'm trying to figure out the relationship between this function and the classical linear regression model.

So, I know that it is a theorem* that we may write

$$ Y = r(X) + \epsilon $$

for some random variable $\epsilon$ s.t. $\mathbb{E}(\epsilon) = 0$.

Now suppose that we have

$$ Y = \beta_0 + \beta_1 X + \epsilon $$

This is the classical 1-dimensional regression function (assuming the $\beta_0$ and $\beta_1$ minimize the residual sum of squares).

Question: Is it then a mathematical theorem that if $Y$ is defined as above, that

$$ r(X) = \mathbb{E}(Y \mid X) = (\beta_0 + \beta_1 X)? $$

And is this why the function $\mathbb{E}(Y \mid X)$ is called the "regression function"?

EDIT: The theorem that I am making use of is as follows (from All of Statistics pg. 89):

Regression models are sometimes written as

$$ Y = r(X) + \epsilon $$

where $\mathbb{E}(\epsilon) = 0$. We can always rewrite a regression model this way. To see this, define $\epsilon = Y - r(X)$ and hence $Y = Y + r(X) - r(X) = r(X) + \epsilon$. Moreover, $\mathbb{E}(\epsilon) = \mathbb{E}\mathbb{E}(\epsilon \mid X) = \mathbb{E}(\mathbb{E}(Y - r(X)) \mid X) = \mathbb{E}(\mathbb{E} ( Y \mid X) - r(X)) = \mathbb{E}(r(X) - r(X)) = 0$.

$\endgroup$
  • 1
    $\begingroup$ The connection is that a linear regression model is exactly the claim that $r$ is a linear function of some observed Xs. Naturally, this claim need not be true, although as an approximation to $r$ it can be better or worse. The chapter of 'Mostly Harmless Econometrics' called 'Making regression make sense' is a good discussion. $\endgroup$ – conjugateprior Jul 3 '16 at 23:34
  • $\begingroup$ Or have I missed what you were asking? $\endgroup$ – conjugateprior Jul 3 '16 at 23:35
  • $\begingroup$ Check the related answer: stats.stackexchange.com/questions/173660/… $\endgroup$ – Tim Jul 4 '16 at 8:20
9
$\begingroup$

Summarizing the question:

Given $Y = \beta_0 + \beta_1 X + \varepsilon$, is it then a mathematical theorem that $r(X) = \mathbb{E}(Y \mid X) = (\beta_0 + \beta_1 X)$?

Yes, by basic properties of expectation:

$$ \begin{align} \operatorname{E}(Y\mid X) & = \operatorname{E}(\beta_0 + \beta_1 X + \varepsilon) \\[6pt] & = \operatorname{E}(\beta_0) + \operatorname{E}(\beta_1 X) + \operatorname{E}(\varepsilon) & & \text{(linearity of expectation)} \\[6pt] & = \beta_0 + \beta_1 X + 0 & & \text{(Noting that $X$ is constant here} \\[-2pt] & & & \quad \text{because we conditioned on it.)} \\[6pt] & = \beta_0 + \beta_1 X \end{align} $$

The historical reasons for regression being called regression relate to Galton noticing the "regression to the mean" effect -- initially in an experiment in plants involving seed-size of offspring compared to the seed size of parents. A relationship through the mean seed size on both variables will have slope less than $1$ (which slope can be estimated by what we call linear regression). The smaller the slope the stronger the "regression" effect. The issue is illustrated by Galton in the linked pdf by heights of children (as adults) compared to average heights of parents (females being scaled up by a constant factor of $8\%$ to make them comparable to males). The diagrams on the third to fifth pages indicate something of what was observed.

So an attempt to estimate the size of this "regression to the mean" is obtained by what came to be called linear regression. Of course there's nothing special going on - the regression to the mean isn't some special biological "drive to mediocrity" as might originally have been guessed, but a fairly simple consequence of the mathematics of the situation in essentially the same sense that correlations are always between $-1$ and $1$.

$\endgroup$
  • $\begingroup$ I've replaced your crude use of \qquad with proper use of "align" in MathJax, plus a few other MathJax details, and I await peer review of the edit. $\qquad$ $\endgroup$ – Michael Hardy Jul 4 '16 at 3:36
  • $\begingroup$ @Michael I'm aware of align and have used it many times -- but what is the actual benefit in the edit in this case? I wanted it left aligned rather than in the center to leave room for the comments to be on a single line and I wanted the comments not to be in the heavy text that MahJax leaves you with, preferring the light text of ordinary markup. The present result is something that no longer matches the appearance I was actually seeking. Rather than being "crude" it was deliberately chosen. If you have a way that achieves what I wanted with less effort than mine required, I'm all ears. $\endgroup$ – Glen_b -Reinstate Monica Jul 4 '16 at 4:37
  • $\begingroup$ ok, I guess not all tastes are in accord with each other$\,\ldots\qquad$ $\endgroup$ – Michael Hardy Jul 4 '16 at 4:43
  • $\begingroup$ The as-designed appearance is ideal, I think, for articles and books but doesn't always reflect what I think is best in a forum like this, at least not always. I recognize my tastes on this issue (and numerous other aspects of site appearance I often try to work around) may be different from the norm, so I'll leave it as it stands, but I don't promise to try to continue to try to get align to do what I want when it seems easier to do it other ways. $\endgroup$ – Glen_b -Reinstate Monica Jul 4 '16 at 5:02
  • $\begingroup$ Does the theorem go the other direction, too? That is, given $X$ and $Y$, can we always conclude that $\mathbb{E}(Y \mid X) = (\beta_0 + \beta_1 X) + \epsilon$ for the $\beta_0, \beta_1$ the regression coefficients? If not, what are the conditions for when we can and cannot say this? $\endgroup$ – George Jul 4 '16 at 10:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.