2
$\begingroup$

The following two results on the residuals ($\epsilon$) in the case of linear regression get stated as assumptions of the linear regressions

  1. $E(\epsilon) = 0$
  2. $cov(X, \epsilon) = 0$

Here is MIT 18.650 professor Philippe Rigollet stating that these are assumptions

However, both of these are just results of fitting a least squares line through any data. Absolutely no assumptions there. You fit $Y = b + aX + \epsilon$ and minimize $(Y - (b + aX))^2$ and the two above stated result fall out naturally

enter image description here

So, how are these assumptions and not necessary results that the residuals ($\epsilon$) would follow when the optimal line is found?

$\endgroup$
7
  • $\begingroup$ In the title you, rightly, spoken about errors but I noted that in the text you skip to residuals. They are not synonym. The assumptions you listed are about errors not residuals. $\endgroup$
    – markowitz
    Feb 20 at 16:12
  • $\begingroup$ The title is not mine. Someone edited it. Isn't $(Y - (b + aX))$ residual? I have read other answers and they state that the error can never be observed and what I wanted to talk about and what the professor derived are the results on the residuals. I have edited the question with the exact screenshot from the lecture $\endgroup$ Feb 21 at 7:23
  • 1
    $\begingroup$ I think I am beginning to see what you guys mean now. In case the model is not correctly specified then the residuals that I am seeing are not the actual noise on top of the DGP and the model parameters are artificially contorted by the modeling process to make the correlation between the residuals and the X 0. But, neither these model parameters nor these residuals are what I am interested in. If my model is correctly specified only then can I say that the error equals the residuals and that the Xs are not correlated with them. Is this the right line of thought? $\endgroup$ Feb 21 at 9:34
  • 1
    $\begingroup$ @figs_and_nuts Exactly, misspecification makes the residuals bad estimates of the errors. I put the emphasis on "estimate" rather than "equals", as a sample is still involved. Also sorry about the confusion, I proposed a title change as before it was not informative. The screenshot alone is unable to tell me about what model are we discussing, and I think this ambiguity is precisely the source of these misunderstandings. $\endgroup$
    – Kuku
    Feb 21 at 11:22
  • 2
    $\begingroup$ Awesome! I learned something today. Thank you Kuku and markowitz. You have my gratitude $\endgroup$ Feb 21 at 11:26

2 Answers 2

4
$\begingroup$

Distinctions

First let us differentiate two levels.

  1. The true model. We can also call this the data generating process or a structural model. It is structural in the sense that it reflects how each variable is structured or generated.
  2. The fitted model. This is independent of how the data came to be generated. This is just a set of rules that receives data as input and gives some statistics as output.

Example

Consider a typical scenario where we have the effect of $X$ on $Y$ but also the presence of an unobserved confounder $U$. So we have:

  • $X \longrightarrow Y$, and
  • $X \longleftarrow U \longrightarrow Y$

Assume for simplicity that all relationships are linear and additive. We could then explicitly write down the true model as:

$$ Y_i = \beta_0 + \beta_1 X_i + \beta_2 U_i + \epsilon_i $$

What tends to be common in econometrics, is not to separate the unobserved confounder from the error term, so they would write the true model as

$$ Y_i = \beta_0 + \beta_1 X_i + \eta_i $$

Where $\eta_i = \beta_2 U_i + \epsilon_i$ for simplicity. Note the regression coefficients are equal only in this simple linear case. So given the true generating process we described (and some additional graphical assumptions), here we know that $\eta$ and $X$ are associated, since $U$ is a cause of $X$ and $U$ is part of $\eta$. Econometricians will say that $X$ is endogenous (in the true model), because $\operatorname{Cov}[X,\eta] \neq 0$. This is why they also say an omitted variable is a source of endogeneity.

Question

You ask

So, how are these assumptions and not necessary results that the residuals ($\epsilon$) would follow when the optimal line is found?

Let's say you fit the model you describe, then you get something like:

$Y_i = \hat{\beta_0} + \hat{\beta_1} X_i + \delta_i$

As you say, by construction due to OLS, $\operatorname{Cov}[X, \delta] = 0$. But, here is the crucial part, here your $\hat{\beta_1}$ is not an estimator for the true $\beta_1$, which would be the causal or structural parameter you care. Because it is assuming things that are not true in the structural model.

In short, what your teacher leaves implicit is that we assume the fitted or empirical model is correctly specified. And this assumption entails that the consequences of our estimating strategy, e.g. OLS, should also apply to the true model, i.e. $\mathbb{E}[\eta] = 0$ and $\operatorname{Cov}[\eta, X] = 0$, which is not the case here.

This is directly related to the point brought by markowitz and the ambiguity existent in Econometrics textbooks relative to these two levels.

$\endgroup$
3
  • 1
    $\begingroup$ Pretty good summary. However It seems me that you can be more parsimonious with notation; and so clearer. In particular you do not need to change parameters notation in the true model with endogenous error, Indeed $\beta_0=\alpha_0$ and $\beta_1=\alpha_1$. $\endgroup$
    – markowitz
    Feb 20 at 13:02
  • 1
    $\begingroup$ Yes, you are right that it is so in this simple linear case, will edit it, thanks! $\endgroup$
    – Kuku
    Feb 20 at 13:25
  • $\begingroup$ Thank you for making it this simple for me to understand :). +1 $\endgroup$ Feb 21 at 7:47
5
$\begingroup$

Why are error properties in linear regression assumptions if they are true by construction?

  1. $E(\epsilon) = 0$
  2. $cov(X, \epsilon) = 0$

... However, both of these are just results of fitting a least squares line through any data. Absolutely no assumptions there.

Your question is about a controversy point in econometrics. If we interpret linear regression like a conditional expectation function in linear form/approximation you are right those two are results true by construction not by assumption. Note that this hold regardless how many data you have, so justifications that swing around difference between residuals and population errors don't work. Read here for more about that:

Regression and the CEF

Definition and delimitation of regression model

However the two above assumpions (or related formulations as some mentioned in the video you added) are treated as key concepts in econometrics, this can only mean that with linear regression Professors have in mind something different from a linear CEF. It seems me that something like a linear structural equation is the what them have in mind (knowingly or not). Read here about the controversy:

How would econometricians answer the objections and recommendations raised by Chen and Pearl (2013)?

This post and refs therein are strongly related:

Under which assumptions a regression can be interpreted causally?

My summary can help you, however the theme is vast and, as just said, controversial ... if you are student I suggest you to follow your teacher.

$\endgroup$
7
  • $\begingroup$ You are correct the theme is really large and I wont be able to follow most of it. However my question much simpler: Professor has: 1. Taken data $(X,y)$ 2. Minimized $(y - (b + aX))^2$ to find $a$ and $b$ 3. Derived the stated two results on the residuals that would hold at the solution found of $a$ and $b$ 4. Called them an assumption. I do not see how that can be. I am trying to follow what the teacher had in mind while saying that $\endgroup$ Feb 20 at 10:16
  • $\begingroup$ Why single out econometrics? The same issues arise with discussing this model in any field of application. $\endgroup$
    – Nick Cox
    Feb 20 at 10:18
  • $\begingroup$ @figs_and_nuts The two assumptions you mention are not about OLS residuals, definitely. I do not heard all the video but Prof. surely do not make so trivial mistake I suppose. Assumptions is about something like "true model" "true errors", the controversy is about the meaning of that. $\endgroup$
    – markowitz
    Feb 20 at 11:16
  • $\begingroup$ @NickCox All that is not about econometrics only. Moreover the delimitations of econometrics and other disciplines is not so clear cut. This controversy emerged in econometrics, I don't know if even elsewhere. $\endgroup$
    – markowitz
    Feb 20 at 11:20
  • 1
    $\begingroup$ Be sure that any good text on regression anywhere discusses what is being assumed -- or postulated -- about the generating process and what is true of residuals given an estimation method. And that many texts give poor or even no serious discussion. For my money, the term assumption itself is profoundly misleading: we'd be better off talking about ideal condtions. $\endgroup$
    – Nick Cox
    Feb 20 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.