Problem Statement

Suppose we have a linear model given by $$y = X\beta + \varepsilon,$$ where $\varepsilon\sim N(0, \sigma^2 I)$ and $E[\varepsilon|X]\neq0$ (i.e., explanatory variables are endogenous). Let the OLS estimate of $\beta$ be denoted $\hat\beta = (X'X)^{-1}X'y$.

One can show that the endogeneity assumption implies $E[\hat\beta]\neq\beta$. However, I've seen the claim that predictions from models suffering from endogeneity remain unbiased. I'm trying to determine if this is true through proof/disproof of the following two claims.

Claim 1: $E[X\hat\beta|X] = X\beta (=E[y|X])$

I'm reasonably sure this is true since OLS chooses $\hat\beta$ to minimize sum of squared residuals. For the proof, I have begun with \begin{align} E[X\hat\beta|X] &= E[X(X'X)^{-1}X'y|X]\\ &=E[X(X'X)^{-1}X'(X\beta + \varepsilon)|X]\\ &=E[X(X'X)^{-1}X'X\beta|X] + E[X(X'X)^{-1}X'\varepsilon|X]\\ &=X\beta + E[X(X'X)^{-1}X'\varepsilon|X]. \end{align} However, I can't determine how the second term is zero since $E[\varepsilon|X]\neq0$.

Claim 2: $E[\hat{X}\hat\beta|X] = \hat{X}\beta$ for $\hat{X}\neq X$

I know this claim isn't true in general. For example, if $\hat{X} = (0, 1, 0, ..., 0)$, then $$E[\hat{X}\hat\beta|X] = E[\hat\beta_1|X] \neq \beta_1$$ in general since $\hat{\beta}$ is not an unbiased estimator of $\beta$.

However, I am interested in sufficient conditions under which this claim is true. For example, is it possible to show this claim is true if $E[X'\varepsilon] = E[\hat{X}'\varepsilon]$ or under some other assumption?

  • $\begingroup$ For claim 1, you can directly show that the expectation is zero by linearity of expectation and the definition of the expectation of a matrix with random variables as entries (i.e. simply write out the expression of the resulting vector) $\endgroup$ – Kevin Li Jul 27 '18 at 19:17
  • $\begingroup$ Hi @KevinLi, thanks for your comment. I'm not seeing exactly what you mean, if you submit an answer explaining further, I'd be happy to accept it. $\endgroup$ – tkmckenzie Jul 27 '18 at 19:27
  • $\begingroup$ Hi, just re-read your question and I was mistaken. OLS can actually have a biased estimate of the regression coefficients if you have endogeneity. $\endgroup$ – Kevin Li Jul 27 '18 at 19:47
  • $\begingroup$ Hi again @KevinLi, I am aware of biased regression coefficients, but I am actually interested in predictions from the regression. I've seen several lectures stating endogeneity should not be a concern if you're only interested in prediction, and I'm looking to validate those claims. $\endgroup$ – tkmckenzie Jul 27 '18 at 19:55
  • 1
    $\begingroup$ +1. This is related to (and partly overlaps with) an unanswered question "T-consistency vs. P-consistency". $\endgroup$ – Richard Hardy Jul 31 '18 at 6:06

The answers are actually pretty straightforward. In general the predictions will not be unbiased, to see that just notice:

$$ E[y|X]= X\beta + E[\epsilon|X] $$

Thus, if $E[\epsilon|X]$ is not linear function of $X$, the population linear regression will not recover the true expectation function $E[y|X]$. Instead, it will give you the best linear approximation of $y$ (best as in minimizing the quadratic error). Since this is true for the population, of course sample estimates are also not consistent.

Analogously, if $E[\epsilon|X]$ is a linear function of $X$, then the population linear regression is by definition $E[y|X]$. So all standard results for linear regression apply, and you will get unbiased predictions. You just won't recover the structural $\beta$, instead you will recover the population regression coefficients $E[XX']^{-1}E[XY]$.


Claims 1 and 2 are actually both false. To show that they are false over a broad class of models in which $E[\epsilon|X] \neq 0$, it is enough to pick a single model or a narrow set of models from within that broad class of models, and to disprove the claims for the narrow set. I will follow this strategy in my answer.

Suppose the true model is $y_i = z_i\Gamma + x_i\beta + \delta_i$ for some zero-mean $\delta_i$'s and for a hidden set of confounders $Z$. Suppose $E[z_i|x_i] = x_iA$. Your $\epsilon_i$ is my $z_i\Gamma + \delta_i = x_iA\Gamma + \delta_i$. Then $E[y_i] = x_iA\Gamma + x_i\beta + \delta_i$, and the OLS estimates are targeting $\tilde\beta \equiv A\Gamma + \beta$ instead of just $\beta$. But, the true function is still linear, so predictions will be unbiased.

To address your claims directly, using my notation:

  • Claim 1 should read $E[ X \hat \beta] = X\tilde\beta$, not $E[X \hat \beta ] = X\beta$. So as stated, it is false. Your second term, in my notation, is $A\Gamma$, which is indeed nonzero.
  • Claim 2 is likewise missing a tilde.

Claim 1 is worth a little more discussion. If it were modified to include $A\Gamma$, it would become true, at least for the limited set of models I have discussed. This is significant because it is a formal statement of the earlier assertion that predictions are unbiased. To expand on this, it's worth stating explicitly what I mean by an unbiased prediction. The predictor that best avoids systematic differences with the true model is $X \beta+ E[\epsilon|X] = X \beta+ E[Z\Gamma + \delta|X] = X \beta+ X A\Gamma$. I would call unbiased any prediction method whose expectation is this. Since $\beta+ A\Gamma$ is exactly what the OLS estimates target, OLS produces unbiased predictions here. (By contrast, this is still biased if you want to infer $\beta$.)

  • $\begingroup$ This is on the right track but only addresses endogeneity caused by omitted variables that are correlated with included explanatory variables in a very specific way (linear relationship between omitted and included variables). Of course, endogeneity is more general than this example, and I'm hoping to prove/disprove these claims in the general case. $\endgroup$ – tkmckenzie Jul 30 '18 at 14:25
  • 1
    $\begingroup$ If claims 1 and 2 are false under a special case of endogeneity, then they are false under any more flexible model of endogeneity. I will edit the answer to clarify this. $\endgroup$ – eric_kernfeld Jul 30 '18 at 15:38
  • $\begingroup$ My answer does not address your last two sentences, which ask for sufficient conditions under which claim 2 holds. I will think about how to improve on this. $\endgroup$ – eric_kernfeld Jul 30 '18 at 15:38
  • $\begingroup$ Your previously submitted answer actually showed the first claim is true in this special case (you still have the sentence "But, the true function is still linear, so predictions will be unbiased."), there was just a difference in our notation. I've confirmed the claim experimentally in this particular case and the general case as well. $\endgroup$ – tkmckenzie Jul 30 '18 at 15:50
  • $\begingroup$ I am not sure if it's the notation, or an error by one of us, or even the definition of the word "unbiased." If you want you claim #1 to say "predictions will be unbiased", then shouldn't it say $E[X\hat \beta] = X \beta + E[\epsilon | X]$? $\endgroup$ – eric_kernfeld Jul 30 '18 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.