6
$\begingroup$

I'm working through an econometrics textbook and it's proving that

$$ \sigma^2 = E(\hat{\sigma}^2) = \frac{SSR}{n-2} $$

I followed the proof (an example of which is shown on talkstats) until it reaches this step:

\begin{align} E\left[\displaystyle\sum\limits_{i=1}^n \hat{u}_{i}^{2}\right] &= E\left[\displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})^{2}\right] + (\hat{\beta}_{1}-\beta_{1})^{2} E\left[\displaystyle\sum\limits_{i=1}^n (x_{i} - \bar{x})^2\right] \\ &+ 2 (\hat{\beta}_{1}-\beta _{1}) E\left[\displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})(x_{i}-\bar{x})\right] \\ &= \dots \end{align}

The book says that:

  1. The expected value of the first term should be $(n-1)\sigma^2$
  2. The expected value of the second term should be $\sigma^2$.
  3. The expected value of the third term should be $2 \sigma^2$ because the term itself can be written as $2(\hat{\beta}_1-\beta_1)^2 \hat{\sigma}_x^2$.

For the first term, I can work backwards and get that:

\begin{align} \hat{\sigma}^2 &= \frac{1}{n-1} \displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})^{2} \\ &= \sigma^2 \end{align}

but it seems like this only works if we assume that $\hat{\sigma}^2$ is an unbiased estimator.

For the second term:

\begin{align} (\hat{\beta}_{1}-\beta_{1})^{2} E\left[\displaystyle\sum\limits_{i=1}^n (x_{i} - \bar{x})^2\right] &= (\hat{\beta}_{1}-\beta_{1})^{2} E\left[(n-1) \hat{\sigma}_x^2\right] \\ &= (\hat{\beta}_{1}-\beta_{1})^{2} (n - 1) E\left[\hat{\sigma}_x^2\right] \\ &= (\hat{\beta}_{1}-\beta_{1})^{2} (n - 1) \hat{\sigma}_x^2 \end{align}

but this doesn't match what the text shows, so I'm a bit confused as to the final step for this term.

For the third term: \begin{align} 2 (\hat{\beta}_{1}-\beta _{1}) E\left[\displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})(x_{i}-\bar{x})\right] &= 2 (\hat{\beta}_{1}-\beta _{1}) E\left[\displaystyle\sum\limits_{i=1}^n u_{i}(x_{i}-\bar{x})\right] \\ &= 2 (\hat{\beta}_{1}-\beta_{1}) \displaystyle\sum\limits_{i=1}^n E\left[u_{i}(x_{i}-\bar{x})\right] \\ &= 2 (\hat{\beta}_{1}-\beta_{1}) \displaystyle\sum\limits_{i=1}^n E\left[u_{i}\right]E\left[(x_{i}-\bar{x})\right] \\ \end{align}

I would have thought that because $E[u_i] = 0$, this term would cancel out, so I'm not sure how to derive $2\sigma^2$ from it.

Can someone give me a hint in the right direction for these steps? I'm firmly trying to work through all of the proofs myself with only a little guidance from the book, but I'm not quite there yet.

I noticed this question, which links to this question, so maybe with more reading I'll see where to apply that. (If that isn't it, hints are welcome, but I'll still work through those questions to see if it helps)

$\endgroup$

2 Answers 2

7
$\begingroup$

I don't work directly through your derivation, but provide a more general formulation below.

For a more general formulation, let your regression model be $Y = X\beta + \epsilon$, $P_X = X(X^\prime X)^{-1} X^\prime$, and $M_X = I_N - P_X$ ($I_N$ is a $N\times N$ identity matrix). $X$ is $N\times K$ and of full column rank. We assume homoskedasticity and no serial correlation.

We show that $\hat{\sigma}^2$ is unbiased:

$$\begin{align*} \mathbb{E}\left[\frac{\hat{\epsilon}^\prime \hat{\epsilon}}{N - K}\mid X\right] &= \mathbb{E}\left[\frac{\epsilon^\prime M^\prime M \epsilon}{N - K}\mid X\right] \\ &= \mathbb{E}\left[\frac{\epsilon^\prime M \epsilon}{N - K}\mid X\right] \\ &= \frac{\sum_{i=1}^N{\sum_{j=1}^N{m_{ji}\mathbb{E}[\epsilon_i\epsilon_j\mid X]}}}{N - K} \\ &= \frac{\sum_{i=1}^N{m_{ii}\sigma^2}}{N - K} \\ &= \frac{\sigma^2\mathop{\text{tr}}(M)}{N - K} \\ \end{align*}$$

$$\begin{align*} \text{tr}(M) &= \text{tr}(I_N - P_X) \\ &= \text{tr}(I_N) - \text{tr}(P_X) \\ &= N - \text{tr}\left(X\left(X^\prime X\right)^{-1}X^\prime\right) \\ &= N - \text{tr}\left(\left(X^\prime X\right)^{-1}X^\prime X\right) \\ &= N - \text{tr}(I_{K}) = N - K \\ \Longrightarrow \mathbb{E}\left[\frac{\hat{\epsilon}^\prime \hat{\epsilon}}{N - K}\mid X\right] &= \frac{\sigma^2 (N-K)}{(N-K)} = \sigma^2. \end{align*}$$

$\endgroup$
3
  • 3
    $\begingroup$ +1. This gets to the geometric essence of the situation: because $M$ is an orthogonal projection whose kernel is precisely the column space of the design matrix, when the errors are iid with variance $\sigma^2$ the expectation of the SSR must equal $N-K$ times $\sigma^2$. However, somewhat contrary to your characterization, we must understand $K$ to be the dimension of the column space (which may be smaller than the second dimension of $X$). Knowing some kind of formula for $M$ is immaterial; these are the only properties needed to obtain the result. $\endgroup$
    – whuber
    Jul 16, 2013 at 15:25
  • $\begingroup$ This is WAY over my head, but I'll take a look at it once I know more. Any hints on the derivation I was using (even though I know its way simpler) $\endgroup$
    – M T
    Jul 17, 2013 at 20:24
  • $\begingroup$ how do you obtain that $\hat \epsilon = M\epsilon$. Can you please give the proof? Thank you. $\endgroup$ Sep 11, 2017 at 21:42
1
$\begingroup$

I think I figured out the version of the proof I was doing even though Charlie's proof is much better (more general I assume).

First term:

\begin{align} E\left[ \displaystyle\sum\limits_{i=1}^n (u_{i}^2-\bar{u})^{2} \right] &= E\left[ \displaystyle\sum\limits_{i=1}^n u_{i}^2-n(\bar{u})^{2} \right] \\ &=E(u_1^2) + \cdots + E(n_n^2) - nE(\bar{u}^2) \\ &= Var(u_1) + E(u_1)E(u_1) + \cdots + Var(u_n) + E(u_n)E(u_n) - n(Var(\bar{u} - E(\bar{u}) E(\bar{u})) \\ &= n\sigma^2 - \frac{1}{n}Var(u_1 + \cdots + u_n) \\ &= n\sigma^2 - \frac{1}{n}[Var(u_1) + \cdots + Var(u_n) ]\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \text{because the $u_i$ are iid}\\ &= n\sigma^2 - \frac{1}{n}[n\sigma^2]\\ &= (n-1)\sigma^2 \end{align}

Second term:

\begin{align} E\left[(\hat{\beta}_{1}-\beta_{1})^{2} \displaystyle\sum\limits_{i=1}^n (x_{i} - \bar{x})^2\right] &= E\left[(\hat{\beta}_{1}-\beta_{1})^{2} s^2_x\right] \\ &= s^2_x E\left[(\hat{\beta}_{1}-\beta_{1})^{2} \right] \\ &= s^2_x \left( Var(\hat{\beta}_{1}-\beta_{1}) + E(\hat{\beta}_{1}-\beta_{1})E(\hat{\beta}_{1}-\beta_{1})\right) \\ &= s^2_x \left( Var(\hat{\beta}_{1}) + 0\right) \\ &= s^2_x \frac{\sigma^2}{s^2_x} \\ &= \sigma^2 \end{align}

I think this works because 1) $E(\hat{\beta}_{1}-\beta_{1}) = 0$ because $\hat{\beta_1}$ is an unbiased estimator of $\beta_1$, and 2) I already proved (when I worked on this question) that $Var(\hat{\beta_1}) = \sigma^2 / s^2_x$.

Third term:

\begin{align} E\left[2 (\hat{\beta}_{1}-\beta _{1}) \displaystyle\sum\limits_{i=1}^n (u_{i}-\bar{u})(x_{i}-\bar{x})\right] &= 2 E\left[(\hat{\beta}_{1}-\beta _{1}) (\hat{\beta}_{1}-\beta _{1}) s^2_x \right] \\ &= 2 s^2_x E\left[(\hat{\beta}_{1}-\beta _{1})^2 \right] \\ &= 2 s^2_x \frac{\sigma^2}{s^2_x} \\ &= 2 \sigma^2 \end{align}

I think this works because it basically just uses the formula I used in this question:

\begin{align} \hat{\beta}_1 - \beta_1 &= \frac{1}{s^2_x} \displaystyle\sum\limits_{i=1}^n (x_i - \bar{x}) u_i \\ \end{align}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.