Prove that the leverage converges to 0. Do the residuals in a linear regression really approximate the errors?

Question

In linear regression, $y_i = x_i^T \beta + \epsilon_i$, $i=1,\dots,n$, and $Var(\epsilon_i)=\sigma^2$. It is well known that the residuals $e_i$ have variance $Var(e_i) = \sigma^2 (1-h_{ii})$, where $h_{ii}$ is the leverage of observation $i$, and that is not equal to $\sigma^2$.

It is commonly believed that the leverage goes to 0, and the residuals $e_i$ approximate the errors $\epsilon_i$. But how can this be justified?

Clearly, it is necessary that the leverage converges to 0 in order for the residuals to approximate the errors, so that they have the same variance asymptotically. But I can find no reference that proves: for all $i=1,\dots,n$, $h_{ii}\to0$ as $n\to\infty$.

It is known that the average of the leverage is $p/n$, where $p$ is the number of parameters, and $h_{ii}>0$. But the mean of a sequence of positive numbers converging to 0, does not imply the elements of the sequence converge to 0. In fact, $h_{ii}$ depends on $x_i$, so it would seem any proof would need to place some assumption on how $x_i$, $i=1,\dots,n$, grows for large $n$.

How does one prove that leverage converges to 0? Is there a reference?

Answer 1

The leverage values are fully determined by the explanatory variables, and since regression analysis imposes no restriction on the explanatory variables, it is not true to say that it is commonly believed that the leverage goes to zero asymptotically.$^\dagger$ To the contrary, asymptotic results in regression analysis generally proceed by showing what happens to the various regression quantities of interest if asymptotic conditions are imposed on the explanatory variables and the resulting leverage values. Typically, the asymptotic results in regression are derived using the "Grenander conditions" (see this related answer) which mean that $\max h_{ii} \rightarrow 0$ as $n \rightarrow \infty$, so that no individual data point is "influential" in the limit. This is imposed as a condition of asymptotic theorems relating to other regression quantities.

Whether or not the maximum leverage of the data points goes to zero asymptotically depends on how you construct the sequence of explanatory variables that you are using for the asymptotic analysis. It is easy to construct sequences of explanatory variables where the maximum leverage does not go to zero, such that there are always "influential" data points, even in the limit.

---

$^\dagger$ When we go beyond this to model the explanatory variables we usually say that we're doing multivariate analysis rather than regression analysis.

Answer 2

Your conjecture is not true. It is probably true when the design matrix $X$ satisfies some regularity conditions, but in general, there is no guarantee that every $h_{ii}$ converges to $0$ (technically, when $X$ is not deterministic, you should also clarify that the convergence mode of $h_{ii}$, in probability or almost surely?).

For example, let $X$ be (it's a deterministic matrix but depends on the sample size $n$): \begin{align} X = \begin{bmatrix} 1 & n \\ 1 & 0 \\ \vdots & \vdots \\ 1 & 0 \end{bmatrix} \in \mathbb{R}^{n \times 2}. \end{align}

It is then easy to verify that $h_{11} = 1$, $h_{ii} = (n - 1)^{-1}$ for $2 \leq i \leq n$. Therefore $h_{11} \not\to 0$ as $n \to \infty$.