How do I use the standard regression assumptions to prove that $\hat{\sigma}^2$ is an unbiased estimator of $\sigma^2$?

Question

I'm working through an econometrics textbook and it's proving that

$$ \sigma^2 = E(\hat{\sigma}^2) = \frac{SSR}{n-2} $$

I followed the proof (an example of which is shown on talkstats) until it reaches this step:

\begin{align} E\left[\displaystyle\sum\limits_{i=1}^n \hat{u}_{i}^{2}\right] &= E\left[\displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})^{2}\right] + (\hat{\beta}_{1}-\beta_{1})^{2} E\left[\displaystyle\sum\limits_{i=1}^n (x_{i} - \bar{x})^2\right] \\ &+ 2 (\hat{\beta}_{1}-\beta _{1}) E\left[\displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})(x_{i}-\bar{x})\right] \\ &= \dots \end{align}

The book says that:

1. The expected value of the first term should be $(n-1)\sigma^2$
2. The expected value of the second term should be $\sigma^2$.
3. The expected value of the third term should be $2 \sigma^2$ because the term itself can be written as $2(\hat{\beta}_1-\beta_1)^2 \hat{\sigma}_x^2$.

For the first term, I can work backwards and get that:

\begin{align} \hat{\sigma}^2 &= \frac{1}{n-1} \displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})^{2} \\ &= \sigma^2 \end{align}

but it seems like this only works if we assume that $\hat{\sigma}^2$ is an unbiased estimator.

For the second term:

\begin{align} (\hat{\beta}_{1}-\beta_{1})^{2} E\left[\displaystyle\sum\limits_{i=1}^n (x_{i} - \bar{x})^2\right] &= (\hat{\beta}_{1}-\beta_{1})^{2} E\left[(n-1) \hat{\sigma}_x^2\right] \\ &= (\hat{\beta}_{1}-\beta_{1})^{2} (n - 1) E\left[\hat{\sigma}_x^2\right] \\ &= (\hat{\beta}_{1}-\beta_{1})^{2} (n - 1) \hat{\sigma}_x^2 \end{align}

but this doesn't match what the text shows, so I'm a bit confused as to the final step for this term.

For the third term: \begin{align} 2 (\hat{\beta}_{1}-\beta _{1}) E\left[\displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})(x_{i}-\bar{x})\right] &= 2 (\hat{\beta}_{1}-\beta _{1}) E\left[\displaystyle\sum\limits_{i=1}^n u_{i}(x_{i}-\bar{x})\right] \\ &= 2 (\hat{\beta}_{1}-\beta_{1}) \displaystyle\sum\limits_{i=1}^n E\left[u_{i}(x_{i}-\bar{x})\right] \\ &= 2 (\hat{\beta}_{1}-\beta_{1}) \displaystyle\sum\limits_{i=1}^n E\left[u_{i}\right]E\left[(x_{i}-\bar{x})\right] \\ \end{align}

I would have thought that because $E[u_i] = 0$, this term would cancel out, so I'm not sure how to derive $2\sigma^2$ from it.

Can someone give me a hint in the right direction for these steps? I'm firmly trying to work through all of the proofs myself with only a little guidance from the book, but I'm not quite there yet.

I noticed this question, which links to this question, so maybe with more reading I'll see where to apply that. (If that isn't it, hints are welcome, but I'll still work through those questions to see if it helps)

Answer 1

I don't work directly through your derivation, but provide a more general formulation below.

For a more general formulation, let your regression model be $Y = X\beta + \epsilon$, $P_X = X(X^\prime X)^{-1} X^\prime$, and $M_X = I_N - P_X$ ($I_N$ is a $N\times N$ identity matrix). $X$ is $N\times K$ and of full column rank. We assume homoskedasticity and no serial correlation.

We show that $\hat{\sigma}^2$ is unbiased:

$$\begin{align*} \mathbb{E}\left[\frac{\hat{\epsilon}^\prime \hat{\epsilon}}{N - K}\mid X\right] &= \mathbb{E}\left[\frac{\epsilon^\prime M^\prime M \epsilon}{N - K}\mid X\right] \\ &= \mathbb{E}\left[\frac{\epsilon^\prime M \epsilon}{N - K}\mid X\right] \\ &= \frac{\sum_{i=1}^N{\sum_{j=1}^N{m_{ji}\mathbb{E}[\epsilon_i\epsilon_j\mid X]}}}{N - K} \\ &= \frac{\sum_{i=1}^N{m_{ii}\sigma^2}}{N - K} \\ &= \frac{\sigma^2\mathop{\text{tr}}(M)}{N - K} \\ \end{align*}$$

$$\begin{align*} \text{tr}(M) &= \text{tr}(I_N - P_X) \\ &= \text{tr}(I_N) - \text{tr}(P_X) \\ &= N - \text{tr}\left(X\left(X^\prime X\right)^{-1}X^\prime\right) \\ &= N - \text{tr}\left(\left(X^\prime X\right)^{-1}X^\prime X\right) \\ &= N - \text{tr}(I_{K}) = N - K \\ \Longrightarrow \mathbb{E}\left[\frac{\hat{\epsilon}^\prime \hat{\epsilon}}{N - K}\mid X\right] &= \frac{\sigma^2 (N-K)}{(N-K)} = \sigma^2. \end{align*}$$

Answer 2

I think I figured out the version of the proof I was doing even though Charlie's proof is much better (more general I assume).

First term:

\begin{align} E\left[ \displaystyle\sum\limits_{i=1}^n (u_{i}^2-\bar{u})^{2} \right] &= E\left[ \displaystyle\sum\limits_{i=1}^n u_{i}^2-n(\bar{u})^{2} \right] \\ &=E(u_1^2) + \cdots + E(n_n^2) - nE(\bar{u}^2) \\ &= Var(u_1) + E(u_1)E(u_1) + \cdots + Var(u_n) + E(u_n)E(u_n) - n(Var(\bar{u} - E(\bar{u}) E(\bar{u})) \\ &= n\sigma^2 - \frac{1}{n}Var(u_1 + \cdots + u_n) \\ &= n\sigma^2 - \frac{1}{n}[Var(u_1) + \cdots + Var(u_n) ]\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{because the $u_i$ are iid}\\ &= n\sigma^2 - \frac{1}{n}[n\sigma^2]\\ &= (n-1)\sigma^2 \end{align}

Second term:

\begin{align} E\left[(\hat{\beta}_{1}-\beta_{1})^{2} \displaystyle\sum\limits_{i=1}^n (x_{i} - \bar{x})^2\right] &= E\left[(\hat{\beta}_{1}-\beta_{1})^{2} s^2_x\right] \\ &= s^2_x E\left[(\hat{\beta}_{1}-\beta_{1})^{2} \right] \\ &= s^2_x \left( Var(\hat{\beta}_{1}-\beta_{1}) + E(\hat{\beta}_{1}-\beta_{1})E(\hat{\beta}_{1}-\beta_{1})\right) \\ &= s^2_x \left( Var(\hat{\beta}_{1}) + 0\right) \\ &= s^2_x \frac{\sigma^2}{s^2_x} \\ &= \sigma^2 \end{align}

I think this works because 1) $E(\hat{\beta}_{1}-\beta_{1}) = 0$ because $\hat{\beta_1}$ is an unbiased estimator of $\beta_1$, and 2) I already proved (when I worked on this question) that $Var(\hat{\beta_1}) = \sigma^2 / s^2_x$.

Third term:

\begin{align} E\left[2 (\hat{\beta}_{1}-\beta _{1}) \displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})(x_{i}-\bar{x})\right] &= 2 E\left[(\hat{\beta}_{1}-\beta _{1}) (\hat{\beta}_{1}-\beta _{1}) s^2_x \right] \\ &= 2 s^2_x E\left[(\hat{\beta}_{1}-\beta _{1})^2 \right] \\ &= 2 s^2_x \frac{\sigma^2}{s^2_x} \\ &= 2 \sigma^2 \end{align}

I think this works because it basically just uses the formula I used in this question:

\begin{align} \hat{\beta}_1 - \beta_1 &= \frac{1}{s^2_x} \displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) u_i \\ \end{align}