# How do I use the standard regression assumptions to prove that $\hat{\sigma}^2$ is an unbiased estimator of $\sigma^2$?

I'm working through an econometrics textbook and it's proving that

$$\sigma^2 = E(\hat{\sigma}^2) = \frac{SSR}{n-2}$$

I followed the proof (an example of which is shown on talkstats) until it reaches this step:

\begin{align} E\left[\displaystyle\sum\limits_{i=1}^n \hat{u}_{i}^{2}\right] &= E\left[\displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})^{2}\right] + (\hat{\beta}_{1}-\beta_{1})^{2} E\left[\displaystyle\sum\limits_{i=1}^n (x_{i} - \bar{x})^2\right] \\ &+ 2 (\hat{\beta}_{1}-\beta _{1}) E\left[\displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})(x_{i}-\bar{x})\right] \\ &= \dots \end{align}

The book says that:

1. The expected value of the first term should be $(n-1)\sigma^2$
2. The expected value of the second term should be $\sigma^2$.
3. The expected value of the third term should be $2 \sigma^2$ because the term itself can be written as $2(\hat{\beta}_1-\beta_1)^2 \hat{\sigma}_x^2$.

For the first term, I can work backwards and get that:

\begin{align} \hat{\sigma}^2 &= \frac{1}{n-1} \displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})^{2} \\ &= \sigma^2 \end{align}

but it seems like this only works if we assume that $\hat{\sigma}^2$ is an unbiased estimator.

For the second term:

\begin{align} (\hat{\beta}_{1}-\beta_{1})^{2} E\left[\displaystyle\sum\limits_{i=1}^n (x_{i} - \bar{x})^2\right] &= (\hat{\beta}_{1}-\beta_{1})^{2} E\left[(n-1) \hat{\sigma}_x^2\right] \\ &= (\hat{\beta}_{1}-\beta_{1})^{2} (n - 1) E\left[\hat{\sigma}_x^2\right] \\ &= (\hat{\beta}_{1}-\beta_{1})^{2} (n - 1) \hat{\sigma}_x^2 \end{align}

but this doesn't match what the text shows, so I'm a bit confused as to the final step for this term.

For the third term: \begin{align} 2 (\hat{\beta}_{1}-\beta _{1}) E\left[\displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})(x_{i}-\bar{x})\right] &= 2 (\hat{\beta}_{1}-\beta _{1}) E\left[\displaystyle\sum\limits_{i=1}^n u_{i}(x_{i}-\bar{x})\right] \\ &= 2 (\hat{\beta}_{1}-\beta_{1}) \displaystyle\sum\limits_{i=1}^n E\left[u_{i}(x_{i}-\bar{x})\right] \\ &= 2 (\hat{\beta}_{1}-\beta_{1}) \displaystyle\sum\limits_{i=1}^n E\left[u_{i}\right]E\left[(x_{i}-\bar{x})\right] \\ \end{align}

I would have thought that because $E[u_i] = 0$, this term would cancel out, so I'm not sure how to derive $2\sigma^2$ from it.

Can someone give me a hint in the right direction for these steps? I'm firmly trying to work through all of the proofs myself with only a little guidance from the book, but I'm not quite there yet.

I noticed this question, which links to this question, so maybe with more reading I'll see where to apply that. (If that isn't it, hints are welcome, but I'll still work through those questions to see if it helps)

I don't work directly through your derivation, but provide a more general formulation below.

For a more general formulation, let your regression model be $Y = X\beta + \epsilon$, $P_X = X(X^\prime X)^{-1} X^\prime$, and $M_X = I_N - P_X$ ($I_N$ is a $N\times N$ identity matrix). $X$ is $N\times K$ and of full column rank. We assume homoskedasticity and no serial correlation.

We show that $\hat{\sigma}^2$ is unbiased:

\begin{align*} \mathbb{E}\left[\frac{\hat{\epsilon}^\prime \hat{\epsilon}}{N - K}\mid X\right] &= \mathbb{E}\left[\frac{\epsilon^\prime M^\prime M \epsilon}{N - K}\mid X\right] \\ &= \mathbb{E}\left[\frac{\epsilon^\prime M \epsilon}{N - K}\mid X\right] \\ &= \frac{\sum_{i=1}^N{\sum_{j=1}^N{m_{ji}\mathbb{E}[\epsilon_i\epsilon_j\mid X]}}}{N - K} \\ &= \frac{\sum_{i=1}^N{m_{ii}\sigma^2}}{N - K} \\ &= \frac{\sigma^2\mathop{\text{tr}}(M)}{N - K} \\ \end{align*}

\begin{align*} \text{tr}(M) &= \text{tr}(I_N - P_X) \\ &= \text{tr}(I_N) - \text{tr}(P_X) \\ &= N - \text{tr}\left(X\left(X^\prime X\right)^{-1}X^\prime\right) \\ &= N - \text{tr}\left(\left(X^\prime X\right)^{-1}X^\prime X\right) \\ &= N - \text{tr}(I_{K}) = N - K \\ \Longrightarrow \mathbb{E}\left[\frac{\hat{\epsilon}^\prime \hat{\epsilon}}{N - K}\mid X\right] &= \frac{\sigma^2 (N-K)}{(N-K)} = \sigma^2. \end{align*}

• +1. This gets to the geometric essence of the situation: because $M$ is an orthogonal projection whose kernel is precisely the column space of the design matrix, when the errors are iid with variance $\sigma^2$ the expectation of the SSR must equal $N-K$ times $\sigma^2$. However, somewhat contrary to your characterization, we must understand $K$ to be the dimension of the column space (which may be smaller than the second dimension of $X$). Knowing some kind of formula for $M$ is immaterial; these are the only properties needed to obtain the result. – whuber Jul 16 '13 at 15:25
• This is WAY over my head, but I'll take a look at it once I know more. Any hints on the derivation I was using (even though I know its way simpler) – M T Jul 17 '13 at 20:24
• how do you obtain that $\hat \epsilon = M\epsilon$. Can you please give the proof? Thank you. – honeybadger Sep 11 '17 at 21:42

I think I figured out the version of the proof I was doing even though Charlie's proof is much better (more general I assume).

First term:

\begin{align} E\left[ \displaystyle\sum\limits_{i=1}^n (u_{i}^2-\bar{u})^{2} \right] &= E\left[ \displaystyle\sum\limits_{i=1}^n u_{i}^2-n(\bar{u})^{2} \right] \\ &=E(u_1^2) + \cdots + E(n_n^2) - nE(\bar{u}^2) \\ &= Var(u_1) + E(u_1)E(u_1) + \cdots + Var(u_n) + E(u_n)E(u_n) - n(Var(\bar{u} - E(\bar{u}) E(\bar{u})) \\ &= n\sigma^2 - \frac{1}{n}Var(u_1 + \cdots + u_n) \\ &= n\sigma^2 - \frac{1}{n}[Var(u_1) + \cdots + Var(u_n) ]\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{because the $u_i$ are iid}\\ &= n\sigma^2 - \frac{1}{n}[n\sigma^2]\\ &= (n-1)\sigma^2 \end{align}

Second term:

\begin{align} E\left[(\hat{\beta}_{1}-\beta_{1})^{2} \displaystyle\sum\limits_{i=1}^n (x_{i} - \bar{x})^2\right] &= E\left[(\hat{\beta}_{1}-\beta_{1})^{2} s^2_x\right] \\ &= s^2_x E\left[(\hat{\beta}_{1}-\beta_{1})^{2} \right] \\ &= s^2_x \left( Var(\hat{\beta}_{1}-\beta_{1}) + E(\hat{\beta}_{1}-\beta_{1})E(\hat{\beta}_{1}-\beta_{1})\right) \\ &= s^2_x \left( Var(\hat{\beta}_{1}) + 0\right) \\ &= s^2_x \frac{\sigma^2}{s^2_x} \\ &= \sigma^2 \end{align}

I think this works because 1) $E(\hat{\beta}_{1}-\beta_{1}) = 0$ because $\hat{\beta_1}$ is an unbiased estimator of $\beta_1$, and 2) I already proved (when I worked on this question) that $Var(\hat{\beta_1}) = \sigma^2 / s^2_x$.

Third term:

\begin{align} E\left[2 (\hat{\beta}_{1}-\beta _{1}) \displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})(x_{i}-\bar{x})\right] &= 2 E\left[(\hat{\beta}_{1}-\beta _{1}) (\hat{\beta}_{1}-\beta _{1}) s^2_x \right] \\ &= 2 s^2_x E\left[(\hat{\beta}_{1}-\beta _{1})^2 \right] \\ &= 2 s^2_x \frac{\sigma^2}{s^2_x} \\ &= 2 \sigma^2 \end{align}

I think this works because it basically just uses the formula I used in this question:

\begin{align} \hat{\beta}_1 - \beta_1 &= \frac{1}{s^2_x} \displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) u_i \\ \end{align}