# Leverage and residuals - Show $\frac{e_i^2}{\| (I - H)Y \|^2} \le 1 - h_{ii}$ where $e_i$ is the $i$-th residual and $h_{ii}$ is leverage

Question: Suppose that $$\boldsymbol{Y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$$, and the errors have zero mean, and are uncorrelated with constant variance. Let $$\hat{\boldsymbol{\beta}}$$ be the least squares estimator of $$\boldsymbol{\beta}$$. Let $$\boldsymbol{H} = \boldsymbol{X}(\boldsymbol{X}^T \boldsymbol{X})^{-1}\boldsymbol{X}^T$$ be the hat matrix, and $$h_{ii}$$ be the $$i$$-th element on its diagonal. Show that $$\frac{e_i^2}{\| (\boldsymbol{I} - \boldsymbol{H})\boldsymbol{Y} \|^2} \le 1 - h_{ii}$$ where $$e_i = Y_i - x_i^T \hat{\boldsymbol{\beta}}$$ is the $$i$$-th residual and $$\boldsymbol{I}$$ is the $$n \times n$$ identity matrix, where $$n$$ is the size of vector $$\boldsymbol{Y}$$.

Attempt: I got as far as writing the $$e_i$$ in terms of the matrices above.

$$e_i = \sum_j (\delta_{ij} - h_{ij})Y_j = Y_i - \sum_j h_{ij}Y_j,$$

where I use $$\boldsymbol{e} = (\boldsymbol{I}-\boldsymbol{H})\boldsymbol{Y}$$. I square to obtain

$$e_i^2 = Y_i^2 - 2 Y_i \sum_j h_{ij}Y_j + (\sum_j h_{ij}Y_j)^2,$$

which is then simplified to

$$e_i^2 = Y_i^2 - 2 Y_i \sum_j h_{ij}Y_j + \sum_j h_{ij}^2 Y_j^2 + \sum_j \sum_{k \neq j} h_{ij} h_{ik} Y_i Y_j.$$

This did not lead me anywhere further. Could anyone please offer me a hint?

Note that, from $$\mathbf e=(\mathbf I-\mathbf H) \mathbf Y,$$ \begin{align}e_i&=-h_{i1}Y_1-h_{i2}Y_2-\cdots+(1-h_{ii})Y_i-\cdots-h_{in}Y_n\\&=\mathbf c^\top\mathbf Y, \tag 1\label 1\end{align} where $$\mathbf c:=(-h_{i1}, h_{i2}, \ldots, (1-h_{ii}), \ldots, -h_{in})^\top.$$

Using the fact that $$\textrm{SS}(\mathbf c^\top\mathbf Y) =\left(\mathbf c^\top\mathbf Y\right)^2/\mathbf c^\top\mathbf c,$$ and $$\mathbf c^\top\mathbf c$$ in $$\eqref 1$$ being equal to $$(1-h_{ii}),$$ the sum of squares due to the $$i$$th residual error component is $$\textrm{SSE}_i=\frac{e_i^2}{(1-h_{ii})}.\tag 2$$

Therefore, the sum of squares for error with $$i$$th case deleted can be given as $$\textrm{SSE}_{[i]}=\textrm{SSE}-\textrm{SSE}_i; \tag 3\label 3$$ (alternatively, you can write \begin{align}\textrm{SSE}_{[i]}&=\mathbf Y_{[i]}^\top\mathbf Y_{[i]}-\mathbf Y_{[i]}^\top\mathbf X_{[i]}\left(\mathbf X_{[i]}^\top\mathbf X_{[i]}\right) ^{-1}\mathbf X_{[i]}^\top\mathbf Y_{[i]}\\&=\left(\mathbf Y^\top\mathbf Y-y_i^2\right) -\left(\mathbf Y^\top\mathbf X-y_i\mathbf x_i^\top\right) \left\{\hat{\boldsymbol{\beta}}-\left[(\mathbf X^\top \mathbf X)^{-1}\mathbf x_ie_i\right]/(1-h_{ii})\right\};\end{align} simplify it and this would yield $$\eqref 3.$$)

$$\textrm{SSE}= \mathbf e^\top\mathbf e=\|\mathbf e\|^2=\|(\mathbf I-\mathbf H) \mathbf Y\|^2$$ and that $$\textrm{SSE}_{[i]}\geq 0;$$ from $$\eqref 3,$$ you can conclude your concerned inequality relation.

## References:

$$\rm [I]$$ Applied Regression Analysis, Norman R. Draper, Harry Smith, John Wiley & Sons, $$1998,$$ pp. $$163, ~207-208.$$

$$\rm [II]$$ Plane Answers to Complex Questions: The Theory of Linear Models, Ronald Christensen, Springer Science$$+$$ Business, $$2011,$$ sec. $$13.6,$$ pp. $$372-373.$$

To save some typing, I will not make symbols boldfaced in the following answer.

In essence, this is just a disguise of Cauchy-Schwarz inequality. To wit, let $$u_i$$ denote the $$n$$-long column vector whose $$i$$-th entry is $$1$$ and all the remaining entries $$0$$. By definition, the inequality of your interest is then (the key observation is that $$1 - h_{ii} = u_i^\top(I - H)u_i$$): \begin{align*} \left(u_i^\top(I - H)Y\right)^2 \leq u_i^\top(I - H)u_i \cdot Y^\top(I - H)Y. \tag{1}\label{zx1} \end{align*} Now use the property that $$I - H$$ is symmetric and idempotent, the expression in the parentheses of the left hand side of $$\eqref{zx1}$$ is $$((I - H)u_i)^\top (I - H)Y$$. Hence $$\eqref{zx1}$$ is equivalent to \begin{align} ((I - H)u_i)^\top(I - H)Y)^2 \leq u_i^\top(I - H)u_i \cdot Y^\top(I - H)Y. \tag{2}\label{zx2} \end{align}

It should be clear now that $$\eqref{zx2}$$ is the Cauchy-Schwarz inequality (the inner-product space form) $$(a^\top b)^2 \leq \|a\|^2 \|b\|^2$$ with $$a = (I - H)u_i$$ and $$b = (I - H)Y$$.

• It should be fine now. I think the culprit is that I used the same "label" as you to reference it. It is indeed a somewhat strange bug to be fixed by stackexchange (before making the label differ from yours, even the equation cannot be rendered) Commented Oct 31, 2023 at 4:44