I am unable to find the bias of the sample variance estimator in this problem. Unfortunately I keep coming up with

Question: Suppose that the true linear model is

$y = X_1\beta_1 + X_2\beta_2 + e$,

where $e \sim MVN(0,\sigma^2 I)$, $X_1: N \times p_1, \beta_1: p_1 \times 1, X_2: N \times p_2, \beta_2: p_2 \times 1$. But we wrongly fit the model $y = X_1\beta_1+e$ and calculate the estimators of $\beta_1$ and $\sigma^2$ as:

$\hat{\beta}_1 = (X_1'X_1)^{-1}X_1'y$,

$s^2 = \large \frac{y'[I-X_1(X_1'X_1)^{-1}X_1']y}{N-p_1}$.

(a) Show that, in general, $\hat{\beta}_1$ is a biased estimator with

Bias$(\hat{\beta}_1) = (X_1'X_1)^{-1}X_1'X_2\hat{\beta}_2$

Under what condition on $X_1$ and $X_2$ is $\hat{\beta}_1$ unbiased?

(b) Show that, in general, $s^2$ is a biased estimator with

Bias$(s^2) = \frac{1}{N-p_1}\beta_2'X_2'[I-X_1(X_1'X_1)^{-1}X_1']X_2\beta_2$

Is the bias positive or negative? Under what condition on $\beta_2$ is $s^2$ unbiased?


For part (a):

Bias$(\hat{\beta}_1) = E((X_1'X_1)^{-1}X_1'y - \beta_1)$

$\hspace{15mm}= (X_1'X_1)^{-1}X_1'E(y) - \beta_1$

$\hspace{15mm}= (X_1'X_1)^{-1}X_1'(X_1\beta_1 + X_2\beta_2) - \beta_1$

$\hspace{15mm}= (X_1'X_1)^{-1}X_1'X_1\beta_1 + (X_1'X_1)^{-1}X_1'X_2\beta_2) - \beta_1$

$\hspace{15mm}= I\beta_1 + (X_1'X_1)^{-1}X_1'X_2\beta_2 - \beta_1$

$\hspace{15mm}= \beta_1 + (X_1'X_1)^{-1}X_1'X_2\beta_2 - \beta_1$

$\hspace{15mm}= (X_1'X_1)^{-1}X_1'X_2\beta_2$

If $X_1$ and $X_2$ are orthogonal, then $\hat{\beta}_1$ is unbiased.

For part (b), I am unable to figure out how to derive:

Bias$(s^2) = \frac{1}{N-p_1}\beta_2'X_2'[I-X_1(X_1'X_1)^{-1}X_1']X_2\beta_2$

I can get


after taking the expectation of y on both sides.

Any help would be much appreciated.


Matrix notation. The sum of squared residuals is

$$\hat e' \hat e = (y-X_1\hat \beta)'(y-X_1\hat \beta) = y'M_1y$$

where $M_1 = I-X_1(X_1'X_1)^{-1}X_1'$, symmetric and idempotent.

Expand $y$ using the true model: $$\hat e' \hat e = (X_1\beta_1 + X_2\beta_2 + e)'M_1(X_1\beta_1 + X_2\beta_2 + e)$$

Now, it is easily verified that $X_1'M_1 = M_1X_1=0$. So we are left with

$$\hat e' \hat e = \beta_2'X_2'M_1X_2\beta_2 +\beta_2'X_2'M_1e+e'M_1X_2\beta_2 +e'M_1e $$

Taking the conditional expected value while using the mean-independence of the error term w.r.t. the regressors, $E(e\mid X_1, X_2) = 0$, we have

$$E(\hat e' \hat e \mid X_1, X_2) = \beta_2'X_2'M_1X_2\beta_2 + E(e'M_1e \mid X_1, X_2)$$

$$\Rightarrow E(\hat e' \hat e \mid X_1, X_2) = \beta_2'X_2'M_1X_2\beta_2 +\sigma^2\cdot \text{trace}(M_1)$$

For the fact that $E(e'M_1e \mid X_1, X_2) = \sigma^2\cdot \text{trace}(M_1)$ consult e.g Hayashi (2000) ch. 1 p. 31, chapter 1 is legally downloadable.

But also (same reference), $\text{trace}(M_1) = N-p_1$ so

$$\Rightarrow E(\hat e' \hat e \mid X_1, X_2) = \beta_2'X_2'M_1X_2\beta_2 +\sigma^2\cdot (N-p_1)$$

The bias of $s^2$ then follows immediately.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.