5
$\begingroup$

Consider the linear regression model: $y_i = \beta + u_i$. We might write this as: $$Y = X\beta + U\text{ with }Y = \begin{pmatrix}y_1 \\ y_2 \\ \vdots \\ y_n\end{pmatrix}, X = \begin{pmatrix}1 \\ 1 \\ \vdots \\ 1\end{pmatrix}, U = \begin{pmatrix}u_1 \\ u_2 \\ \vdots \\ u_n\end{pmatrix}$$ Then, I want to find the asymptotic distribution of the estimated error variance: $$\hat{\sigma}_u^2 = \frac{1}{n-1}\sum_{i=1}^n \hat{u}_i^2 = \frac{\hat{U}'\hat{U}}{n-1}$$ where $\hat{U} = Y - X\hat{\beta}$ and $\hat{\beta} = (X'X)^{-1}X'Y$. Then: $$\hat{U} = Y - X(X'X)^{-1}X'Y = (I_n - X(X'X)^{-1}X')Y := MY = MX\beta + MU = (X\beta - X(X'X)^{-1}X'X\beta) + MU = MU$$ where $M$ is symmetric ($M = M'$) and idempotent ($MM = M$). Then: $$\hat{\sigma}_u^2 = \frac{\hat{U}'\hat{U}}{n-1} = \frac{U'MU}{n-1} = \frac{U'U}{n-1} - \frac{U'X(X'X)^{-1}X'U}{n-1} = \frac{U'U}{n-1} - (\hat{\beta} - \beta)\frac{X'U}{n-1}$$ So: $$\hat{\sigma}_u^2 - \frac{U'U}{n-1} = - (\hat{\beta} - \beta)\frac{X'U}{n-1}$$ $$\sqrt{n}(\hat{\sigma}_u^2 - \frac{U'U}{n-1}) = -\sqrt{n}(\hat{\beta} - \beta)\frac{X'U}{n-1}$$ and since as $n \rightarrow \infty$, $\frac{1}{n-1} \rightarrow \frac{1}{n}$, then by the Lindeberg-Levy LLN for $iid$ random variables, $\frac{U'U}{n-1} \xrightarrow{p} \sigma_u^2$. Then, by the Lindeberg-Levy CLT: $$\sqrt{n}(\hat{\beta} - \beta) \xrightarrow{d} N(0, \sigma_u^2)$$ So: $$\sqrt{n}(\hat{\sigma}_u^2 - \sigma_u^2) \xrightarrow{d} \frac{U'X}{n-1} Z \text{ where }Z\sim N(0, \sigma_u^2)$$ Now, the expected value and variance of $X'U$ is: $$\mathbb{E}(X'U) = \mathbb{E}(X'\mathbb{E}[U|X]) = 0$$ $$Var(X'U) = \mathbb{E}((X'U)(X'U)') = \mathbb{E}(X'\mathbb{E}[UU'|X]X) = \sigma_u^2 \mathbb{E}(X'X) = n\sigma_u^2$$ This last step is since $X'X = \sum_{i=1}^n X_i^2 = n$, so $AVar(\frac{X'U}{n-1}) = \sigma_u^2$. My issue is I'm not entirely sure how to bring these results together. A lecturer mentioned that the eventual result is: $$\sqrt{n}(\hat{\sigma}_u^2 - \sigma_u^2) \xrightarrow{d} N(0,(\mu_4 - \sigma_u^4)^4)$$ But I don't see how to reach this point.

$\endgroup$

1 Answer 1

6
$\begingroup$

Let me start by giving you a hint, and if that is enough, then no need to read through the rest of the solution.

Hint: $\beta=(X'X)^{-1}(X'Y)=n^{-1}\sum_i^nY_i=\bar Y$. So the sample variance is given by,

$$\hat\sigma^2=\sum_{i=1}^n(Y_i - X_i \beta)^2=\sum_{i=1}^n(Y_i - \bar Y)^2$$


Solution:

Okay, if that hint is not enough here is a solution. I will omit some algebra so you might have some verification to do on your own. Notice, there is a couple of ways to accomplish this but here is my preferred solution.

Note that,

$$\hat\sigma^2 = \frac{1}{n-1}\sum_{i=1}^n(Y_i - \bar Y)^2 = \frac{n}{n-1}(\sum_i Y_i^2)-(2n\bar Y n^{-1}\sum_i Y_i)+n\bar Y^2 = \frac{n}{n-1}(\frac{1}{n}(Y_i^2-\bar Y^2)=\frac{n}{n-1}(\bar{Y^2}-\bar Y^2)$$

Note that $\bar{Y^2}:=n^{-1}\sum_i (Y_i^2)$.

Now we will use the delta-method with $h(a,b)=b-a^2$. So that we have,

$$h(\bar Y, \bar{Y^2} ) = \bar{Y^2} - (\bar Y)^2=\frac{n}{n-1}\hat\sigma^2$$

and

$$h(E[Y], E[Y^2] ) = E[Y^2] - E[Y]^2=\sigma^2$$

Then we start with,

$$\sqrt{n}(h(\bar Y, \bar{Y^2}) - h(E[Y], E[Y^2]))\overset{d}{\to} \nabla h(E[Y], E[Y^2])\cdot N(0,V) = N(0, (\nabla h) V (\nabla h)^T)$$

(Note that we should first check the joint normality with the MVT CLT this follows from the fact that $Y_i$ is iid.)

So all that is left is to identify this covariance matrix. Recall,

$$V = \begin{bmatrix}Var[Y]& Cov[Y^2, Y]\\Cov[Y^2, Y]&Var[Y^2]\end{bmatrix}$$

So

$$(\nabla h )V (\nabla h)^T=\begin{bmatrix}-2E[Y] & 1\end{bmatrix} V \begin{bmatrix}-2E[Y] \\ 1\end{bmatrix} \\= \begin{bmatrix}-2E[Y]Var[Y] + Cov[Y^2, Y] & -2E[Y]Cov[Y^2 Y]+Var[Y^2]\end{bmatrix}\begin{bmatrix}-2E[Y] \\ 1\end{bmatrix} \\= -2E[Y](-2E[Y]Var[Y] + Cov[Y^2, Y])-2E[Y]Cov[Y, Y^2] + Var[Y^2]$$

Before we proceed we need to compute some variances, $$E[Y^2]=Var[Y]+E[Y]^2$$ $$Cov(Y, Y^2)=E[(Y-E[Y])(Y^2-E[Y^2])]=E[Y^3]-E[Y]E[Y^2]=E[Y^3]-\mu(\sigma^2+\mu^2)$$

$$Var[Y^2]=E[Y^4]-E[Y^2]^2 = E[Y^4]-(\sigma^2+\mu^2)^2$$

So plugging in and simplifying we eventually should get,

$$\nabla h V \nabla h^T= E[(Y-\mu)^4]-\sigma^4$$

So putting this together (using Slutsky's theorem and noticing $\frac{n}{n-1} \overset{p}{\to} 1$) we get,

$$\sqrt{n}(\hat\sigma^2-\sigma^2)\overset{d}{\to} N(0,E[(Y-\mu)^4]-\sigma^4)$$

Or setting $\mu_4=E[(Y-\mu)^4]$ (the centered fourth moment) we get,

$$\sqrt{n}(\hat\sigma^2-\sigma^2)\overset{d}{\to} N(0,\mu_4-\sigma^4)$$

Which is what you have (I think the extra power of 4 is a typo?).

Also, a nice way to motivate this formula is to realize that it is exactly equal to $Var[(Y-\mu)^2]=E[(Y-\mu)^4]-E[(Y-\mu)^2]^2=\mu_4-\sigma^4$. Which is great because if we knew $\mu$ our variance estimator would be $n^{-1}\sum_i (Y_i-\mu)^2$.

$\endgroup$
4
  • 2
    $\begingroup$ You can use >! to hide text behind an interactive "reveal spoiler" grey wall if you don't want the full solution to be immediately visible. $\endgroup$
    – microhaus
    Jun 1, 2021 at 21:39
  • $\begingroup$ Oh cool, I will try that out thanks! $\endgroup$
    – Ariel
    Jun 1, 2021 at 21:45
  • $\begingroup$ Hi, thanks for this! Just as a follow-up, do you know of any good resources on how to extend this approach to cases where $\beta \neq \bar{Y}$? So where $X$ is more interesting than just an intercept term? $\endgroup$
    – greggs
    Jun 2, 2021 at 21:14
  • 1
    $\begingroup$ @greggs it should be possible to apply a similar technique for this case. The trick will be dealing with a more complicated covariance matrix, $V$, when showing joint normality. I am guessing you will have to deal with the variance-covariance of terms like $\bar{Y^2}$, $\bar{X^2}$, and $\bar{XY}$ when doing this. $\endgroup$
    – Ariel
    Jun 2, 2021 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.