2
$\begingroup$

As argued in this answer, we can compute the residuals in the standard linear regression: \begin{align} y &= X\hat{\beta} + e,\\ e &= y - X(X^\top X)^{-1}X^\top y = (I-H)y, \end{align} where $H=X(X^\top X)^{-1}X^\top$ is the projection matrix. We can derive the conditional variance of the residual $e$ as $Var(e\mid X)=Var((I-H)y\mid X)=Var((I-H)\epsilon\mid X)=\sigma^2 (I-H)(I-H)^\top=\sigma^2 (I-H).$

As $n\to \infty$, we can expect $Var(e\mid X) \stackrel{d}{\to} \mathcal{N}(0, \sigma^2 I)$, as the residual errors will reflect the error distribution assumed by the model. which would mean $H\to 0$. This may require the assumption of Grenander conditions, $\frac 1n\mathbf X'\mathbf X\stackrel{p}{\to} \mathbf{Q}$, where $\mathbf{Q}$ is a positive definite matrix. But how does this condition ensure $H\to 0$?

$\endgroup$
2
  • $\begingroup$ Hi: you should probably put a hat on your $e$ just to show that they are the estimates. But, AFAIK, I don't think they converge to normal zero mean and variance $\sigma^2$ because it's not the case that $H \longrightarrow 0$. I'm confident that one of the gurus on this list will correct me if I'm wrong which could be the case. $\endgroup$
    – mlofton
    Aug 6 at 5:18
  • $\begingroup$ The link here is not an exact answer but it doesn't sound like they converge to anything.stats.stackexchange.com/questions/22468/… $\endgroup$
    – mlofton
    Aug 7 at 1:30

1 Answer 1

4
$\begingroup$

For a matrix to converge to the zero matrix, every element of it must converge to zero, under one of the criteria suitable for random variables. For a $n \times K$ regressor matrix ${\rm X}$, with typical row $\mathbf x_j' = (x_{1j}, x_{2j},...,x_{Kj})$ a single element of the associated "hat" matrix $H$, say $h_{ji}$ can be written as

$$h_{ji} = \mathbf x_j'({\rm X}'{\rm X})^{-1}\mathbf x_{i} = \frac 1n \mathbf x_j'\left(\frac 1n{\rm X}'{\rm X}\right)^{-1}\mathbf x_{i}. $$

Part of the assumptions of such models is that

$$\left(\frac 1n{\rm X}'{\rm X}\right)^{-1} \rightarrow_p Q < \infty.$$

So it appears we get

$${\rm plim }\, h_{ji} = {\rm plim } \left( \frac 1n \mathbf x_j'Q\mathbf x_{i}\right)$$

which converges to zero since $\mathbf x_j'Q\mathbf x_{i}$ is a finite number.

Don't confuse this with the result that

$$\sum_{i=1}^n h_{ii} = K.$$

This says that the sum of the diagonal elements equals the number of regressor always, and even when the sample size goes to infinity. But if all elements go to zero, how can this be?

Well, it is not something unique to the projection matrix. Let $I_n$ be the $n \times n$ Identity matrix and consider that $${\rm lim} \frac 1n I_n \to \mathbf 0_n.$$

But the sum of diagonal elements will always be equal to 1. Yes, "understanding" how it is possible that every element viewed on its own goes to zero but their sum is always strictly positive may be difficult, but when infinity appears, intuition is strained.

Another issue: if $H \to \mathbf 0$ what happens to the vector of predictions $\hat y = Hy$? Does it also go to zero? No because $$Hy = X(X'X)^{-1}X'(X\beta + \varepsilon) = X\beta + H\varepsilon,$$

namely, the $H$ matrix remains, already for finite $n$, only in connection to the error term.

$\endgroup$
3
  • $\begingroup$ @Bravo This part has now been corrected. $\endgroup$ Aug 8 at 0:00
  • $\begingroup$ Good, so the predictions $\mathbf{\hat{y}}\to X\beta$ w.p 1. $\endgroup$
    – Bravo
    Aug 8 at 8:55
  • $\begingroup$ @Bravo Indeed it does. $\endgroup$ Aug 8 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.