Consider the linear regression model: $y_i = \beta + u_i$. We might write this as: $$Y = X\beta + U\text{ with }Y = \begin{pmatrix}y_1 \\ y_2 \\ \vdots \\ y_n\end{pmatrix}, X = \begin{pmatrix}1 \\ 1 \\ \vdots \\ 1\end{pmatrix}, U = \begin{pmatrix}u_1 \\ u_2 \\ \vdots \\ u_n\end{pmatrix}$$ Then, I want to find the asymptotic distribution of the estimated error variance: $$\hat{\sigma}_u^2 = \frac{1}{n-1}\sum_{i=1}^n \hat{u}_i^2 = \frac{\hat{U}'\hat{U}}{n-1}$$ where $\hat{U} = Y - X\hat{\beta}$ and $\hat{\beta} = (X'X)^{-1}X'Y$. Then: $$\hat{U} = Y - X(X'X)^{-1}X'Y = (I_n - X(X'X)^{-1}X')Y := MY = MX\beta + MU = (X\beta - X(X'X)^{-1}X'X\beta) + MU = MU$$ where $M$ is symmetric ($M = M'$) and idempotent ($MM = M$). Then: $$\hat{\sigma}_u^2 = \frac{\hat{U}'\hat{U}}{n-1} = \frac{U'MU}{n-1} = \frac{U'U}{n-1} - \frac{U'X(X'X)^{-1}X'U}{n-1} = \frac{U'U}{n-1} - (\hat{\beta} - \beta)\frac{X'U}{n-1}$$ So: $$\hat{\sigma}_u^2 - \frac{U'U}{n-1} = - (\hat{\beta} - \beta)\frac{X'U}{n-1}$$ $$\sqrt{n}(\hat{\sigma}_u^2 - \frac{U'U}{n-1}) = -\sqrt{n}(\hat{\beta} - \beta)\frac{X'U}{n-1}$$ and since as $n \rightarrow \infty$, $\frac{1}{n-1} \rightarrow \frac{1}{n}$, then by the Lindeberg-Levy LLN for $iid$ random variables, $\frac{U'U}{n-1} \xrightarrow{p} \sigma_u^2$. Then, by the Lindeberg-Levy CLT: $$\sqrt{n}(\hat{\beta} - \beta) \xrightarrow{d} N(0, \sigma_u^2)$$ So: $$\sqrt{n}(\hat{\sigma}_u^2 - \sigma_u^2) \xrightarrow{d} \frac{U'X}{n-1} Z \text{ where }Z\sim N(0, \sigma_u^2)$$ Now, the expected value and variance of $X'U$ is: $$\mathbb{E}(X'U) = \mathbb{E}(X'\mathbb{E}[U|X]) = 0$$ $$Var(X'U) = \mathbb{E}((X'U)(X'U)') = \mathbb{E}(X'\mathbb{E}[UU'|X]X) = \sigma_u^2 \mathbb{E}(X'X) = n\sigma_u^2$$ This last step is since $X'X = \sum_{i=1}^n X_i^2 = n$, so $AVar(\frac{X'U}{n-1}) = \sigma_u^2$. My issue is I'm not entirely sure how to bring these results together. A lecturer mentioned that the eventual result is: $$\sqrt{n}(\hat{\sigma}_u^2 - \sigma_u^2) \xrightarrow{d} N(0,(\mu_4 - \sigma_u^4)^4)$$ But I don't see how to reach this point.
1 Answer
Let me start by giving you a hint, and if that is enough, then no need to read through the rest of the solution.
Hint: $\beta=(X'X)^{-1}(X'Y)=n^{-1}\sum_i^nY_i=\bar Y$. So the sample variance is given by,
$$\hat\sigma^2=\sum_{i=1}^n(Y_i - X_i \beta)^2=\sum_{i=1}^n(Y_i - \bar Y)^2$$
Solution:
Okay, if that hint is not enough here is a solution. I will omit some algebra so you might have some verification to do on your own. Notice, there is a couple of ways to accomplish this but here is my preferred solution.
Note that,
$$\hat\sigma^2 = \frac{1}{n-1}\sum_{i=1}^n(Y_i - \bar Y)^2 = \frac{n}{n-1}(\sum_i Y_i^2)-(2n\bar Y n^{-1}\sum_i Y_i)+n\bar Y^2 = \frac{n}{n-1}(\frac{1}{n}(Y_i^2-\bar Y^2)=\frac{n}{n-1}(\bar{Y^2}-\bar Y^2)$$
Note that $\bar{Y^2}:=n^{-1}\sum_i (Y_i^2)$.
Now we will use the delta-method with $h(a,b)=b-a^2$. So that we have,
$$h(\bar Y, \bar{Y^2} ) = \bar{Y^2} - (\bar Y)^2=\frac{n}{n-1}\hat\sigma^2$$
and
$$h(E[Y], E[Y^2] ) = E[Y^2] - E[Y]^2=\sigma^2$$
Then we start with,
$$\sqrt{n}(h(\bar Y, \bar{Y^2}) - h(E[Y], E[Y^2]))\overset{d}{\to} \nabla h(E[Y], E[Y^2])\cdot N(0,V) = N(0, (\nabla h) V (\nabla h)^T)$$
(Note that we should first check the joint normality with the MVT CLT this follows from the fact that $Y_i$ is iid.)
So all that is left is to identify this covariance matrix. Recall,
$$V = \begin{bmatrix}Var[Y]& Cov[Y^2, Y]\\Cov[Y^2, Y]&Var[Y^2]\end{bmatrix}$$
So
$$(\nabla h )V (\nabla h)^T=\begin{bmatrix}-2E[Y] & 1\end{bmatrix} V \begin{bmatrix}-2E[Y] \\ 1\end{bmatrix} \\= \begin{bmatrix}-2E[Y]Var[Y] + Cov[Y^2, Y] & -2E[Y]Cov[Y^2 Y]+Var[Y^2]\end{bmatrix}\begin{bmatrix}-2E[Y] \\ 1\end{bmatrix} \\= -2E[Y](-2E[Y]Var[Y] + Cov[Y^2, Y])-2E[Y]Cov[Y, Y^2] + Var[Y^2]$$
Before we proceed we need to compute some variances, $$E[Y^2]=Var[Y]+E[Y]^2$$ $$Cov(Y, Y^2)=E[(Y-E[Y])(Y^2-E[Y^2])]=E[Y^3]-E[Y]E[Y^2]=E[Y^3]-\mu(\sigma^2+\mu^2)$$
$$Var[Y^2]=E[Y^4]-E[Y^2]^2 = E[Y^4]-(\sigma^2+\mu^2)^2$$
So plugging in and simplifying we eventually should get,
$$\nabla h V \nabla h^T= E[(Y-\mu)^4]-\sigma^4$$
So putting this together (using Slutsky's theorem and noticing $\frac{n}{n-1} \overset{p}{\to} 1$) we get,
$$\sqrt{n}(\hat\sigma^2-\sigma^2)\overset{d}{\to} N(0,E[(Y-\mu)^4]-\sigma^4)$$
Or setting $\mu_4=E[(Y-\mu)^4]$ (the centered fourth moment) we get,
$$\sqrt{n}(\hat\sigma^2-\sigma^2)\overset{d}{\to} N(0,\mu_4-\sigma^4)$$
Which is what you have (I think the extra power of 4 is a typo?).
Also, a nice way to motivate this formula is to realize that it is exactly equal to $Var[(Y-\mu)^2]=E[(Y-\mu)^4]-E[(Y-\mu)^2]^2=\mu_4-\sigma^4$. Which is great because if we knew $\mu$ our variance estimator would be $n^{-1}\sum_i (Y_i-\mu)^2$.
-
2$\begingroup$ You can use
>!
to hide text behind an interactive "reveal spoiler" grey wall if you don't want the full solution to be immediately visible. $\endgroup$ Jun 1, 2021 at 21:39 -
-
$\begingroup$ Hi, thanks for this! Just as a follow-up, do you know of any good resources on how to extend this approach to cases where $\beta \neq \bar{Y}$? So where $X$ is more interesting than just an intercept term? $\endgroup$– greggsJun 2, 2021 at 21:14
-
1$\begingroup$ @greggs it should be possible to apply a similar technique for this case. The trick will be dealing with a more complicated covariance matrix, $V$, when showing joint normality. I am guessing you will have to deal with the variance-covariance of terms like $\bar{Y^2}$, $\bar{X^2}$, and $\bar{XY}$ when doing this. $\endgroup$– ArielJun 2, 2021 at 22:13