I would be grateful if you could help me clear up some confusion regarding conditional expectation and regression. I have seen two formulations of the linear regression framework: $$Y=a+bX+\varepsilon\qquad\qquad(1)$$ and $$\mathbb{E}[Y|X]=a+bX.\qquad\qquad(2)$$ Often, in an introduction to linear regression it is said that linearity of the relationship (between $X$ and $Y$) is assumed. For example, on Wikipedia it reads "...a linear regression model assumes that the relationship between the dependent variable y and the vector of regressors x is linear." However, in my experience, in any given application linear regression is rather seen as a linear approximation, which makes much more sense to me than assuming a truly linear relationship between $Y$ and $X$ (i.e., draw $X$, transform it linearly and add some random noise $\varepsilon$), but I have not seen this mentioned in introductory textbooks (and maybe this is wrong). I'm wondering why that is and how (1) and (2) can be related.

More specifically, my confusion is the following. In general, approximations can be seen as orthogonal projections. For example, for any r.v. $Y\in L^2$, the conditional expectation is the orthogonal projection on the space of (equivalence classes of) $X$-measurable r.v.s (i.e., functions of $X$ because of factorization). Therefore, it minimizes the MSE and we can write $$Y=\mathbb{E}[Y|X]+\varepsilon_1,\qquad\varepsilon_1:=Y-\mathbb{E}[Y|X].\qquad\qquad(3)$$ On the other hand, because the orthogonal projection is the unique minimizer of the MSE, if $a$ and $b$ in (1) are obtained by least squares, $a+bX$ is the orthogonal projection of $Y$ on the space of linear functions of $X$. But then how does the conditional expectation come into play here? Where is it gone?

Here is the explanation I can come up with, maybe you can tell me if this makes sense. We would like to have (3), because we want the best approximation (for example to make predictions) of $Y$ given $X$. Moreover, we would like to have (2) because linear functions are nice. Unfortunately, in reality, except maybe if $X,Y$ are (approximately) jointly normal, we have neither. However, if we find the best $X$-linear approximation to $Y$, we simultaneously find the best $X$-linear approximation linear approximation to $\mathbb{E}[Y|X]$ (right? Because $\parallel\mathbb{E}[Y|X]-L(X)\parallel\leq\parallel\mathbb{E}[Y|X]-Y\parallel+\parallel Y-L(X)\parallel$ for $L$ linear is minimized by the best approximation $a+bX$ to $Y$), that is, $$\mathbb{E}[Y|X]=a+bX+\varepsilon_2,\qquad\varepsilon_2:=\mathbb{E}[Y|X]-(a+bX).\qquad(4)$$ In other words, we project twice, that is $L^2\overset{p_1}{\to}\{f(X)\}\overset{p_2}{\to}\{\tilde{a}+\tilde{b}X\}$.

This also means that in the error in (1) there are actually two errors, that is, \begin{align} Y&=\mathbb{E}[Y|X]+\varepsilon_1\\ &=a+bX+\varepsilon_2+\varepsilon_1\\ &=a+bX+\varepsilon, \end{align} with $\varepsilon=\varepsilon_1+\varepsilon_2$, but of course they are not seperable.

My questions are:

  • Does that make sense?
  • If yes, can you point me to a reference that discusses this explicitly in the social sciences, preferably a textbook? I found this question, where they reference Hansen (2022) who basically describes linear regression as I did here (except for the double projection) and this question that is very similar, but I have never seen in it the social sciences and I'm still confused for example by the Wikipedia quote (and related ones from other textbooks). What is going on here?
  • One last technical question: What if we find $a$ and $b$ not by the least squares but by some other algorithm (e.g., WLS, RML). Then we don't do orthogonal projection (right? Because it's unique?), so the error is not uncorrelated with $X$. Is that bad? What do we do?

References: Hansen, B. E. (2022). Econometrics. Princeton University Press.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.