Do residuals in linear regression converge to $\mathcal{N}(0, \sigma^2 I)$?

Question

As argued in this answer, we can compute the residuals in the standard linear regression: \begin{align} y &= X\hat{\beta} + e,\\ e &= y - X(X^\top X)^{-1}X^\top y = (I-H)y, \end{align} where $H=X(X^\top X)^{-1}X^\top$ is the projection matrix. We can derive the conditional variance of the residual $e$ as $Var(e\mid X)=Var((I-H)y\mid X)=Var((I-H)\epsilon\mid X)=\sigma^2 (I-H)(I-H)^\top=\sigma^2 (I-H).$

As $n\to \infty$, we can expect $Var(e\mid X) \stackrel{d}{\to} \mathcal{N}(0, \sigma^2 I)$, as the residual errors will reflect the error distribution assumed by the model. which would mean $H\to 0$. This may require the assumption of Grenander conditions, $\frac 1n\mathbf X'\mathbf X\stackrel{p}{\to} \mathbf{Q}$, where $\mathbf{Q}$ is a positive definite matrix. But how does this condition ensure $H\to 0$?

Answer 1

For a matrix to converge to the zero matrix, every element of it must converge to zero, under one of the criteria suitable for random variables. For a $n \times K$ regressor matrix ${\rm X}$, with typical row $\mathbf x_j' = (x_{1j}, x_{2j},...,x_{Kj})$ a single element of the associated "hat" matrix $H$, say $h_{ji}$ can be written as

$$h_{ji} = \mathbf x_j'({\rm X}'{\rm X})^{-1}\mathbf x_{i} = \frac 1n \mathbf x_j'\left(\frac 1n{\rm X}'{\rm X}\right)^{-1}\mathbf x_{i}. $$

Part of the assumptions of such models is that

$$\left(\frac 1n{\rm X}'{\rm X}\right)^{-1} \rightarrow_p Q < \infty.$$

So it appears we get

$${\rm plim }\, h_{ji} = {\rm plim } \left( \frac 1n \mathbf x_j'Q\mathbf x_{i}\right)$$

which converges to zero since $\mathbf x_j'Q\mathbf x_{i}$ is a finite number.

Don't confuse this with the result that

$$\sum_{i=1}^n h_{ii} = K.$$

This says that the sum of the diagonal elements equals the number of regressor always, and even when the sample size goes to infinity. But if all elements go to zero, how can this be?

Well, it is not something unique to the projection matrix. Let $I_n$ be the $n \times n$ Identity matrix and consider that $${\rm lim} \frac 1n I_n \to \mathbf 0_n.$$

But the sum of diagonal elements will always be equal to 1. Yes, "understanding" how it is possible that every element viewed on its own goes to zero but their sum is always strictly positive may be difficult, but when infinity appears, intuition is strained.

Another issue: if $H \to \mathbf 0$ what happens to the vector of predictions $\hat y = Hy$? Does it also go to zero? No because $$Hy = X(X'X)^{-1}X'(X\beta + \varepsilon) = X\beta + H\varepsilon,$$

namely, the $H$ matrix remains, already for finite $n$, only in connection to the error term.