# Proof that predictions are unbiased in in endogenous linear model

## Problem Statement

Suppose we have a linear model given by $$y = X\beta + \varepsilon,$$ where $\varepsilon\sim N(0, \sigma^2 I)$ and $E[\varepsilon|X]\neq0$ (i.e., explanatory variables are endogenous). Let the OLS estimate of $\beta$ be denoted $\hat\beta = (X'X)^{-1}X'y$.

One can show that the endogeneity assumption implies $E[\hat\beta]\neq\beta$. However, I've seen the claim that predictions from models suffering from endogeneity remain unbiased. I'm trying to determine if this is true through proof/disproof of the following two claims.

## Claim 1: $E[X\hat\beta|X] = X\beta (=E[y|X])$

I'm reasonably sure this is true since OLS chooses $\hat\beta$ to minimize sum of squared residuals. For the proof, I have begun with \begin{align} E[X\hat\beta|X] &= E[X(X'X)^{-1}X'y|X]\\ &=E[X(X'X)^{-1}X'(X\beta + \varepsilon)|X]\\ &=E[X(X'X)^{-1}X'X\beta|X] + E[X(X'X)^{-1}X'\varepsilon|X]\\ &=X\beta + E[X(X'X)^{-1}X'\varepsilon|X]. \end{align} However, I can't determine how the second term is zero since $E[\varepsilon|X]\neq0$.

## Claim 2: $E[\hat{X}\hat\beta|X] = \hat{X}\beta$ for $\hat{X}\neq X$

I know this claim isn't true in general. For example, if $\hat{X} = (0, 1, 0, ..., 0)$, then $$E[\hat{X}\hat\beta|X] = E[\hat\beta_1|X] \neq \beta_1$$ in general since $\hat{\beta}$ is not an unbiased estimator of $\beta$.

However, I am interested in sufficient conditions under which this claim is true. For example, is it possible to show this claim is true if $E[X'\varepsilon] = E[\hat{X}'\varepsilon]$ or under some other assumption?

• For claim 1, you can directly show that the expectation is zero by linearity of expectation and the definition of the expectation of a matrix with random variables as entries (i.e. simply write out the expression of the resulting vector) – Kevin Li Jul 27 '18 at 19:17
• Hi @KevinLi, thanks for your comment. I'm not seeing exactly what you mean, if you submit an answer explaining further, I'd be happy to accept it. – tkmckenzie Jul 27 '18 at 19:27
• Hi, just re-read your question and I was mistaken. OLS can actually have a biased estimate of the regression coefficients if you have endogeneity. – Kevin Li Jul 27 '18 at 19:47
• Hi again @KevinLi, I am aware of biased regression coefficients, but I am actually interested in predictions from the regression. I've seen several lectures stating endogeneity should not be a concern if you're only interested in prediction, and I'm looking to validate those claims. – tkmckenzie Jul 27 '18 at 19:55
• +1. This is related to (and partly overlaps with) an unanswered question "T-consistency vs. P-consistency". – Richard Hardy Jul 31 '18 at 6:06

The answers are actually pretty straightforward. In general the predictions will not be unbiased, to see that just notice:

$$E[y|X]= X\beta + E[\epsilon|X]$$

Thus, if $$E[\epsilon|X]$$ is not linear function of $$X$$, the population linear regression will not recover the true expectation function $$E[y|X]$$. Instead, it will give you the best linear approximation of $$y$$ (best as in minimizing the quadratic error). Since this is true for the population, of course sample estimates are also not consistent.

Analogously, if $$E[\epsilon|X]$$ is a linear function of $$X$$, then the population linear regression is by definition $$E[y|X]$$. So all standard results for linear regression apply, and you will get unbiased predictions. You just won't recover the structural $$\beta$$, instead you will recover the population regression coefficients $$E[XX']^{-1}E[XY]$$.

Claims 1 and 2 are actually both false. To show that they are false over a broad class of models in which $E[\epsilon|X] \neq 0$, it is enough to pick a single model or a narrow set of models from within that broad class of models, and to disprove the claims for the narrow set. I will follow this strategy in my answer.

Suppose the true model is $y_i = z_i\Gamma + x_i\beta + \delta_i$ for some zero-mean $\delta_i$'s and for a hidden set of confounders $Z$. Suppose $E[z_i|x_i] = x_iA$. Your $\epsilon_i$ is my $z_i\Gamma + \delta_i = x_iA\Gamma + \delta_i$. Then $E[y_i] = x_iA\Gamma + x_i\beta + \delta_i$, and the OLS estimates are targeting $\tilde\beta \equiv A\Gamma + \beta$ instead of just $\beta$. But, the true function is still linear, so predictions will be unbiased.

• Claim 1 should read $E[ X \hat \beta] = X\tilde\beta$, not $E[X \hat \beta ] = X\beta$. So as stated, it is false. Your second term, in my notation, is $A\Gamma$, which is indeed nonzero.
Claim 1 is worth a little more discussion. If it were modified to include $A\Gamma$, it would become true, at least for the limited set of models I have discussed. This is significant because it is a formal statement of the earlier assertion that predictions are unbiased. To expand on this, it's worth stating explicitly what I mean by an unbiased prediction. The predictor that best avoids systematic differences with the true model is $X \beta+ E[\epsilon|X] = X \beta+ E[Z\Gamma + \delta|X] = X \beta+ X A\Gamma$. I would call unbiased any prediction method whose expectation is this. Since $\beta+ A\Gamma$ is exactly what the OLS estimates target, OLS produces unbiased predictions here. (By contrast, this is still biased if you want to infer $\beta$.)
• I am not sure if it's the notation, or an error by one of us, or even the definition of the word "unbiased." If you want you claim #1 to say "predictions will be unbiased", then shouldn't it say $E[X\hat \beta] = X \beta + E[\epsilon | X]$? – eric_kernfeld Jul 30 '18 at 19:22