Asymptotic distribution of OLS standard errors

Question

Consider the linear regression model: $y_i = \beta + u_i$. We might write this as: $$Y = X\beta + U\text{ with }Y = \begin{pmatrix}y_1 \\ y_2 \\ \vdots \\ y_n\end{pmatrix}, X = \begin{pmatrix}1 \\ 1 \\ \vdots \\ 1\end{pmatrix}, U = \begin{pmatrix}u_1 \\ u_2 \\ \vdots \\ u_n\end{pmatrix}$$ Then, I want to find the asymptotic distribution of the estimated error variance: $$\hat{\sigma}_u^2 = \frac{1}{n-1}\sum_{i=1}^n \hat{u}_i^2 = \frac{\hat{U}'\hat{U}}{n-1}$$ where $\hat{U} = Y - X\hat{\beta}$ and $\hat{\beta} = (X'X)^{-1}X'Y$. Then: $$\hat{U} = Y - X(X'X)^{-1}X'Y = (I_n - X(X'X)^{-1}X')Y := MY = MX\beta + MU = (X\beta - X(X'X)^{-1}X'X\beta) + MU = MU$$ where $M$ is symmetric ($M = M'$) and idempotent ($MM = M$). Then: $$\hat{\sigma}_u^2 = \frac{\hat{U}'\hat{U}}{n-1} = \frac{U'MU}{n-1} = \frac{U'U}{n-1} - \frac{U'X(X'X)^{-1}X'U}{n-1} = \frac{U'U}{n-1} - (\hat{\beta} - \beta)\frac{X'U}{n-1}$$ So: $$\hat{\sigma}_u^2 - \frac{U'U}{n-1} = - (\hat{\beta} - \beta)\frac{X'U}{n-1}$$ $$\sqrt{n}(\hat{\sigma}_u^2 - \frac{U'U}{n-1}) = -\sqrt{n}(\hat{\beta} - \beta)\frac{X'U}{n-1}$$ and since as $n \rightarrow \infty$, $\frac{1}{n-1} \rightarrow \frac{1}{n}$, then by the Lindeberg-Levy LLN for $iid$ random variables, $\frac{U'U}{n-1} \xrightarrow{p} \sigma_u^2$. Then, by the Lindeberg-Levy CLT: $$\sqrt{n}(\hat{\beta} - \beta) \xrightarrow{d} N(0, \sigma_u^2)$$ So: $$\sqrt{n}(\hat{\sigma}_u^2 - \sigma_u^2) \xrightarrow{d} \frac{U'X}{n-1} Z \text{ where }Z\sim N(0, \sigma_u^2)$$ Now, the expected value and variance of $X'U$ is: $$\mathbb{E}(X'U) = \mathbb{E}(X'\mathbb{E}[U|X]) = 0$$ $$Var(X'U) = \mathbb{E}((X'U)(X'U)') = \mathbb{E}(X'\mathbb{E}[UU'|X]X) = \sigma_u^2 \mathbb{E}(X'X) = n\sigma_u^2$$ This last step is since $X'X = \sum_{i=1}^n X_i^2 = n$, so $AVar(\frac{X'U}{n-1}) = \sigma_u^2$. My issue is I'm not entirely sure how to bring these results together. A lecturer mentioned that the eventual result is: $$\sqrt{n}(\hat{\sigma}_u^2 - \sigma_u^2) \xrightarrow{d} N(0,(\mu_4 - \sigma_u^4)^4)$$ But I don't see how to reach this point.

Answer 1

Let me start by giving you a hint, and if that is enough, then no need to read through the rest of the solution.

Hint: $\beta=(X'X)^{-1}(X'Y)=n^{-1}\sum_i^nY_i=\bar Y$. So the sample variance is given by,

$$\hat\sigma^2=\sum_{i=1}^n(Y_i - X_i \beta)^2=\sum_{i=1}^n(Y_i - \bar Y)^2$$

---

Solution:

Okay, if that hint is not enough here is a solution. I will omit some algebra so you might have some verification to do on your own. Notice, there is a couple of ways to accomplish this but here is my preferred solution.

Note that,

$$\hat\sigma^2 = \frac{1}{n-1}\sum_{i=1}^n(Y_i - \bar Y)^2 = \frac{n}{n-1}(\sum_i Y_i^2)-(2n\bar Y n^{-1}\sum_i Y_i)+n\bar Y^2 = \frac{n}{n-1}(\frac{1}{n}(Y_i^2-\bar Y^2)=\frac{n}{n-1}(\bar{Y^2}-\bar Y^2)$$

Note that $\bar{Y^2}:=n^{-1}\sum_i (Y_i^2)$.

Now we will use the delta-method with $h(a,b)=b-a^2$. So that we have,

$$h(\bar Y, \bar{Y^2} ) = \bar{Y^2} - (\bar Y)^2=\frac{n}{n-1}\hat\sigma^2$$

and

$$h(E[Y], E[Y^2] ) = E[Y^2] - E[Y]^2=\sigma^2$$

Then we start with,

$$\sqrt{n}(h(\bar Y, \bar{Y^2}) - h(E[Y], E[Y^2]))\overset{d}{\to} \nabla h(E[Y], E[Y^2])\cdot N(0,V) = N(0, (\nabla h) V (\nabla h)^T)$$

(Note that we should first check the joint normality with the MVT CLT this follows from the fact that $Y_i$ is iid.)

So all that is left is to identify this covariance matrix. Recall,

$$V = \begin{bmatrix}Var[Y]& Cov[Y^2, Y]\\Cov[Y^2, Y]&Var[Y^2]\end{bmatrix}$$

So

$$(\nabla h )V (\nabla h)^T=\begin{bmatrix}-2E[Y] & 1\end{bmatrix} V \begin{bmatrix}-2E[Y] \\ 1\end{bmatrix} \\= \begin{bmatrix}-2E[Y]Var[Y] + Cov[Y^2, Y] & -2E[Y]Cov[Y^2 Y]+Var[Y^2]\end{bmatrix}\begin{bmatrix}-2E[Y] \\ 1\end{bmatrix} \\= -2E[Y](-2E[Y]Var[Y] + Cov[Y^2, Y])-2E[Y]Cov[Y, Y^2] + Var[Y^2]$$

Before we proceed we need to compute some variances, $$E[Y^2]=Var[Y]+E[Y]^2$$ $$Cov(Y, Y^2)=E[(Y-E[Y])(Y^2-E[Y^2])]=E[Y^3]-E[Y]E[Y^2]=E[Y^3]-\mu(\sigma^2+\mu^2)$$

$$Var[Y^2]=E[Y^4]-E[Y^2]^2 = E[Y^4]-(\sigma^2+\mu^2)^2$$

So plugging in and simplifying we eventually should get,

$$\nabla h V \nabla h^T= E[(Y-\mu)^4]-\sigma^4$$

So putting this together (using Slutsky's theorem and noticing $\frac{n}{n-1} \overset{p}{\to} 1$) we get,

$$\sqrt{n}(\hat\sigma^2-\sigma^2)\overset{d}{\to} N(0,E[(Y-\mu)^4]-\sigma^4)$$

Or setting $\mu_4=E[(Y-\mu)^4]$ (the centered fourth moment) we get,

$$\sqrt{n}(\hat\sigma^2-\sigma^2)\overset{d}{\to} N(0,\mu_4-\sigma^4)$$

Which is what you have (I think the extra power of 4 is a typo?).

Also, a nice way to motivate this formula is to realize that it is exactly equal to $Var[(Y-\mu)^2]=E[(Y-\mu)^4]-E[(Y-\mu)^2]^2=\mu_4-\sigma^4$. Which is great because if we knew $\mu$ our variance estimator would be $n^{-1}\sum_i (Y_i-\mu)^2$.