# Asymptotic distribution of OLS standard errors

Consider the linear regression model: $$y_i = \beta + u_i$$. We might write this as: $$Y = X\beta + U\text{ with }Y = \begin{pmatrix}y_1 \\ y_2 \\ \vdots \\ y_n\end{pmatrix}, X = \begin{pmatrix}1 \\ 1 \\ \vdots \\ 1\end{pmatrix}, U = \begin{pmatrix}u_1 \\ u_2 \\ \vdots \\ u_n\end{pmatrix}$$ Then, I want to find the asymptotic distribution of the estimated error variance: $$\hat{\sigma}_u^2 = \frac{1}{n-1}\sum_{i=1}^n \hat{u}_i^2 = \frac{\hat{U}'\hat{U}}{n-1}$$ where $$\hat{U} = Y - X\hat{\beta}$$ and $$\hat{\beta} = (X'X)^{-1}X'Y$$. Then: $$\hat{U} = Y - X(X'X)^{-1}X'Y = (I_n - X(X'X)^{-1}X')Y := MY = MX\beta + MU = (X\beta - X(X'X)^{-1}X'X\beta) + MU = MU$$ where $$M$$ is symmetric ($$M = M'$$) and idempotent ($$MM = M$$). Then: $$\hat{\sigma}_u^2 = \frac{\hat{U}'\hat{U}}{n-1} = \frac{U'MU}{n-1} = \frac{U'U}{n-1} - \frac{U'X(X'X)^{-1}X'U}{n-1} = \frac{U'U}{n-1} - (\hat{\beta} - \beta)\frac{X'U}{n-1}$$ So: $$\hat{\sigma}_u^2 - \frac{U'U}{n-1} = - (\hat{\beta} - \beta)\frac{X'U}{n-1}$$ $$\sqrt{n}(\hat{\sigma}_u^2 - \frac{U'U}{n-1}) = -\sqrt{n}(\hat{\beta} - \beta)\frac{X'U}{n-1}$$ and since as $$n \rightarrow \infty$$, $$\frac{1}{n-1} \rightarrow \frac{1}{n}$$, then by the Lindeberg-Levy LLN for $$iid$$ random variables, $$\frac{U'U}{n-1} \xrightarrow{p} \sigma_u^2$$. Then, by the Lindeberg-Levy CLT: $$\sqrt{n}(\hat{\beta} - \beta) \xrightarrow{d} N(0, \sigma_u^2)$$ So: $$\sqrt{n}(\hat{\sigma}_u^2 - \sigma_u^2) \xrightarrow{d} \frac{U'X}{n-1} Z \text{ where }Z\sim N(0, \sigma_u^2)$$ Now, the expected value and variance of $$X'U$$ is: $$\mathbb{E}(X'U) = \mathbb{E}(X'\mathbb{E}[U|X]) = 0$$ $$Var(X'U) = \mathbb{E}((X'U)(X'U)') = \mathbb{E}(X'\mathbb{E}[UU'|X]X) = \sigma_u^2 \mathbb{E}(X'X) = n\sigma_u^2$$ This last step is since $$X'X = \sum_{i=1}^n X_i^2 = n$$, so $$AVar(\frac{X'U}{n-1}) = \sigma_u^2$$. My issue is I'm not entirely sure how to bring these results together. A lecturer mentioned that the eventual result is: $$\sqrt{n}(\hat{\sigma}_u^2 - \sigma_u^2) \xrightarrow{d} N(0,(\mu_4 - \sigma_u^4)^4)$$ But I don't see how to reach this point.

Let me start by giving you a hint, and if that is enough, then no need to read through the rest of the solution.

Hint: $$\beta=(X'X)^{-1}(X'Y)=n^{-1}\sum_i^nY_i=\bar Y$$. So the sample variance is given by,

$$\hat\sigma^2=\sum_{i=1}^n(Y_i - X_i \beta)^2=\sum_{i=1}^n(Y_i - \bar Y)^2$$

Solution:

Okay, if that hint is not enough here is a solution. I will omit some algebra so you might have some verification to do on your own. Notice, there is a couple of ways to accomplish this but here is my preferred solution.

Note that,

$$\hat\sigma^2 = \frac{1}{n-1}\sum_{i=1}^n(Y_i - \bar Y)^2 = \frac{n}{n-1}(\sum_i Y_i^2)-(2n\bar Y n^{-1}\sum_i Y_i)+n\bar Y^2 = \frac{n}{n-1}(\frac{1}{n}(Y_i^2-\bar Y^2)=\frac{n}{n-1}(\bar{Y^2}-\bar Y^2)$$

Note that $$\bar{Y^2}:=n^{-1}\sum_i (Y_i^2)$$.

Now we will use the delta-method with $$h(a,b)=b-a^2$$. So that we have,

$$h(\bar Y, \bar{Y^2} ) = \bar{Y^2} - (\bar Y)^2=\frac{n}{n-1}\hat\sigma^2$$

and

$$h(E[Y], E[Y^2] ) = E[Y^2] - E[Y]^2=\sigma^2$$

$$\sqrt{n}(h(\bar Y, \bar{Y^2}) - h(E[Y], E[Y^2]))\overset{d}{\to} \nabla h(E[Y], E[Y^2])\cdot N(0,V) = N(0, (\nabla h) V (\nabla h)^T)$$

(Note that we should first check the joint normality with the MVT CLT this follows from the fact that $$Y_i$$ is iid.)

So all that is left is to identify this covariance matrix. Recall,

$$V = \begin{bmatrix}Var[Y]& Cov[Y^2, Y]\\Cov[Y^2, Y]&Var[Y^2]\end{bmatrix}$$

So

$$(\nabla h )V (\nabla h)^T=\begin{bmatrix}-2E[Y] & 1\end{bmatrix} V \begin{bmatrix}-2E[Y] \\ 1\end{bmatrix} \\= \begin{bmatrix}-2E[Y]Var[Y] + Cov[Y^2, Y] & -2E[Y]Cov[Y^2 Y]+Var[Y^2]\end{bmatrix}\begin{bmatrix}-2E[Y] \\ 1\end{bmatrix} \\= -2E[Y](-2E[Y]Var[Y] + Cov[Y^2, Y])-2E[Y]Cov[Y, Y^2] + Var[Y^2]$$

Before we proceed we need to compute some variances, $$E[Y^2]=Var[Y]+E[Y]^2$$ $$Cov(Y, Y^2)=E[(Y-E[Y])(Y^2-E[Y^2])]=E[Y^3]-E[Y]E[Y^2]=E[Y^3]-\mu(\sigma^2+\mu^2)$$

$$Var[Y^2]=E[Y^4]-E[Y^2]^2 = E[Y^4]-(\sigma^2+\mu^2)^2$$

So plugging in and simplifying we eventually should get,

$$\nabla h V \nabla h^T= E[(Y-\mu)^4]-\sigma^4$$

So putting this together (using Slutsky's theorem and noticing $$\frac{n}{n-1} \overset{p}{\to} 1$$) we get,

$$\sqrt{n}(\hat\sigma^2-\sigma^2)\overset{d}{\to} N(0,E[(Y-\mu)^4]-\sigma^4)$$

Or setting $$\mu_4=E[(Y-\mu)^4]$$ (the centered fourth moment) we get,

$$\sqrt{n}(\hat\sigma^2-\sigma^2)\overset{d}{\to} N(0,\mu_4-\sigma^4)$$

Which is what you have (I think the extra power of 4 is a typo?).

Also, a nice way to motivate this formula is to realize that it is exactly equal to $$Var[(Y-\mu)^2]=E[(Y-\mu)^4]-E[(Y-\mu)^2]^2=\mu_4-\sigma^4$$. Which is great because if we knew $$\mu$$ our variance estimator would be $$n^{-1}\sum_i (Y_i-\mu)^2$$.

• You can use >! to hide text behind an interactive "reveal spoiler" grey wall if you don't want the full solution to be immediately visible. Jun 1, 2021 at 21:39
• Oh cool, I will try that out thanks! Jun 1, 2021 at 21:45
• Hi, thanks for this! Just as a follow-up, do you know of any good resources on how to extend this approach to cases where $\beta \neq \bar{Y}$? So where $X$ is more interesting than just an intercept term? Jun 2, 2021 at 21:14
• @greggs it should be possible to apply a similar technique for this case. The trick will be dealing with a more complicated covariance matrix, $V$, when showing joint normality. I am guessing you will have to deal with the variance-covariance of terms like $\bar{Y^2}$, $\bar{X^2}$, and $\bar{XY}$ when doing this. Jun 2, 2021 at 22:13