Diagonal elements of the projection matrix

I am having some problem trying to prove that the diagonal elements of the hat matrix $$h_{ii}$$ are between $$1/n$$ and $$1$$.

Suppose that $$\mathrm{Range}(X_{n,k})=K$$ the number of columns of our matrix of data with a constant.$$\implies H_{k,k}$$

$$H=X(X' X)^{-1}X' \implies H'=H ;H^{2}=H$$

If $$y = \beta x + \epsilon \implies HY =\hat Y; (I-H)=\epsilon$$

$$\mathbf{H}=\begin{bmatrix}h_{11} &... &h_{1n}\\⋮ & ⋱ &⋮\\ h_{n1} & ... & h_{nn}\end{bmatrix}$$

If $$\boldsymbol 1=(1, \ldots, 1) \in X \implies 1H^2=1H=1 \implies \sum h_{i1}^2=\sum h_{i1}=1 , h_{ii}\leq 1$$

So $$∑h_{i1}^2\sum h_{i2}^2...\sum h_{in}^2=1 \implies h_{11}^2h_{22}^2...h_{nn}^2\leq 1$$

How to prove that $$h_{ii} \geq(1/n)$$?

(exercise 3.4 from Meyer "Classical and modern regression with applications") Let $$h_{ii}$$ be the ith diagonal of the Hat matrix H. (a) prove that for a multiple regression model with a constant term, $$h_{ii} \geq(1/n)$$ ; (b) show that $$h_{ii}\leq 1$$ (Hint: make the use of the fact that H is idempotent)

• The result is not generally true: the diagonal elements can be less than $1/n$ when $X'X$ is not of full rank (and the generalized inverse is used).
– whuber
Jun 17, 2013 at 14:33
• $H$, as a projection matrix, is rarely of full rank. Even when $X'X$ is invertible, your conclusion is incorrect. Consider $X=(1,2)'$, where $X'X=(5)$, $(X'X)^{-1}=(1/5)$, and $H=X(X'X)^{-1}X'=\left( \begin{array}{cc} \frac{1}{5} & \frac{2}{5} \\ \frac{2}{5} & \frac{4}{5} \end{array} \right)$ has a diagonal entry less than $1/n=1/2$.
– whuber
Jun 17, 2013 at 14:47
• Please don't put your assumptions into comments: edit the question to include all the assumptions you wish to make.
– whuber
Jun 17, 2013 at 14:52
• $H$ will be of full rank only when $X$ is square: in that case you will no longer be doing least squares, but merely solving a completely determined set of linear equations. In general, the rank of $H$ does not exceed the number of columns of the design matrix $X$. In my counterexample, $X$ has one column and the rank of $H$ is one: as large as possible.
– whuber
Jun 17, 2013 at 16:53
• @whuber you are right my apollogies. When I said full range I mean that the range(X)= the number of regressor Jun 17, 2013 at 17:33

This is several years later, but I found the notation very difficult in the asker's question and self-answer, so here's a cleaner solution.

We have $$\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$$ where $$(1,...,1)^T$$ is a column of $$\mathbf{X}$$. We want to show that the diagonals $$h_{ii}$$ of $$\mathbf{H}$$ have $$h_{ii} \geq 1/n$$.

Define $$\mathbf{P} = \mathbf{H} - \mathbf{C}$$, where

$$\mathbf{C} = \frac{1}{n}\begin{pmatrix}1 & \dots & 1 \\ \vdots & \ddots & \vdots \\ 1 & \dots & 1 \end{pmatrix}$$

the matrix consisting of only $$1/n$$. This is the projection matrix onto the space spanned by $$(1, ..., 1)$$. Then

$$\mathbf{P}^2 = \mathbf{H}^2 - \mathbf{H}\mathbf{C} - \mathbf{C}\mathbf{H} + \mathbf{C}^2 = \mathbf{H} - \mathbf{H}\mathbf{C} - \mathbf{C}\mathbf{H} + \mathbf{C}$$

However, $$\mathbf{H}$$ orthogonally projects onto $$\text{Col}(\mathbf{X})$$, and $$\mathbf{C}$$ orthogonally projects onto $$\text{span}\{(1,...,1)\} \subset \text{Col}(\mathbf{X})$$, so obviously $$\mathbf{H}\mathbf{C} = \mathbf{C}$$. Still intuitively, but less obviously, $$\mathbf{C}\mathbf{H} = \mathbf{C}$$. To see this, we can compute $$\mathbf{C} = \mathbf{C}\big(\mathbf{H} + (\mathbf{I} - \mathbf{H})\big)$$, and note that $$\mathbf{C}(\mathbf{I} - \mathbf{H}) = 0$$ because $$\mathbf{I} - \mathbf{H}$$ projects onto $$\text{Col}(\mathbf{X})^\perp$$.

Therefore we have $$\mathbf{P}^2 = \mathbf{H} - \mathbf{C} = \mathbf{P}$$. So $$\mathbf{P}$$ is also a projection matrix.

So $$h_{ii} = p_{ii} + c_{ii} = p_{ii} + 1/n$$. Since projection matrices are always positive semidefinite, the diagonals of $$\mathbf{P}$$ satisfy $$p_{ii} \geq 0$$. (In fact, you can show that since $$\mathbf{P}$$ is symmetric and idempotent, it satisfies $$0 \leq p_{ii} \leq 1$$.)

Then $$h_{ii} \geq 1/n$$ as needed.

• Could you explain how it follows that "$P$ is also a projection matrix"?
– whuber
Apr 24, 2019 at 2:44
• Thanks, I glossed over that at first. Clarification now added. Apr 24, 2019 at 15:14
• Thank you: I felt it was important to make that explicit connection with the assumption that the constant vector is in the span of the columns of $X.$
– whuber
Apr 24, 2019 at 16:26
• What means of $\mathbf{Col(X)}$? Nov 11, 2021 at 9:28
• @aminroshani It should be the space spanned by the column vectors of $\mathbf{X}$. Sep 15, 2022 at 13:05

For prove that $$h_{ii} \geq (1/n)$$, we can center $$H_c=X(X_c' X_c)^{-1}X_c'$$ , $$\mathbf{H_c}=\begin{bmatrix}x_{11}-\bar x_1 &... &x_{1n}-\bar x_1 \\⋮ & ⋱ &⋮\\ x_{n1}-\bar x_n & ... & x_{nn}-\bar x_n\end{bmatrix}$$

$$y=\alpha1+ X_c'\beta +\epsilon⇒ \hat y=\hat \alpha1+ X_c'\hat\beta ⇒ \hat y=\bar y+ X_c'\hat\beta= \bar y+ X_c'(X_c' X_c)^{-1}X_c'y⇒ \hat y=[(1/n) 1'y]1+H_cy$$ $$=[1/n\begin{bmatrix}1&... &1\\⋮ & ⋱ &⋮\\1 & ... & 1\end{bmatrix}+H_c ] y=Hy$$

Then $$H=1/n\begin{bmatrix}1&... &1\\⋮ & ⋱ &⋮\\1 & ... & 1\end{bmatrix}+H_c$$$$h_{ii} \geq (1/n)$$ because $$H_c$$ is a positive definite matrix.

Here is another answer that that only uses the fact that all the eigenvalues of a symmetric idempotent matrix are at most 1, see one of the previous answers or prove it yourself, it's quite easy.

Let $$H$$ denote the hat matrix. The $$i$$th diagonal element of the hat matrix is given by

$$h_{ii} = \mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i,$$

where $$\mathbf{e}_i^{t}$$ is the vector whose $$i$$th element is 1 and the rest are 0s. Consider the quadratic form on the unit sphere given by

$$f(\mathbf{x}) = \frac{\mathbf{x}^{t} \mathbf{H} \mathbf{x}}{\mathbf{x}^{t} \mathbf{x}}.$$

It is well known that the maximum of this expression is $$\lambda_n$$, the largest eigenvalue of the matrix $$\mathbf{H}$$. Returning to the diagonal elements of the hat matrix, one therefore has

$$h_{ii} = \mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i = \frac{\mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i}{\mathbf{e}_i^{t} \mathbf{e}_i} \underbrace{\mathbf{e}_i^{t} \mathbf{e}_i}_{ = 1} \leq \lambda_n \leq1$$

and this gives us what we need.

• How does this argument help prove $h_{ii} \geq \frac{1}{n}$? Dec 30, 2022 at 1:57

Unlike the clever "centering" proof given by Drew N, the proof below is kind of brutal-force (but it has the advantage of giving even a sharper lower bound than $$n^{-1}$$, see $$(1)$$).

Partition the design matrix $$X \in \mathbb{R}^{n \times p}$$ to be $$X = \begin{bmatrix} e & Z\end{bmatrix}$$, where $$e$$ is the intercept term consisting of $$n$$ ones, $$Z$$ is the matrix consisting of $$X$$'s remaining $$p - 1$$ columns. By block matrix inversion formula, \begin{align} & (X'X)^{-1} = \begin{bmatrix} n & e'Z \\ Z'e & Z'Z \end{bmatrix}^{-1} \\ =& \begin{bmatrix} n^{-1} + n^{-2}e'Z(Z'PZ)^{-1}Z'e & -n^{-1}e'Z(Z'PZ)^{-1} \\ -n^{-1}(Z'PZ)^{-1}Z'e & (Z'PZ)^{-1} \end{bmatrix}, \end{align} where $$P = I_{(n)} - n^{-1}ee'$$ is idempotent, hence $$A := Z'PZ$$ is invertible (hence positive definite. The invertibility of $$A$$ is proved in detail at the end). It then follows by $$h_{ii} = x_i'(X'X)^{-1}x_i$$ that \begin{align} & h_{ii} = \begin{bmatrix} 1 & z_i' \end{bmatrix} \begin{bmatrix} n^{-1} + n^{-2}e'ZA^{-1}Z'e & -n^{-1}e'ZA^{-1} \\ -n^{-1}A^{-1}Z'e & A^{-1} \end{bmatrix} \begin{bmatrix} 1 \\ z_i \end{bmatrix} \\ =& \begin{bmatrix} n^{-1} + n^{-2}e'ZA^{-1}Z'e - n^{-1}z_i'A^{-1}Z'e & - n^{-1}e'ZA^{-1} + z_i'A^{-1} \end{bmatrix} \begin{bmatrix} 1 \\ z_i \end{bmatrix} \\ =& n^{-1} + n^{-2}e'ZA^{-1}Z'e - n^{-1}z_i'A^{-1}Z'e - n^{-1}e'ZA^{-1}z_i + z_i'A^{-1}z_i \\ \geq & n^{-1} + \left(n^{-1}\sqrt{e'ZA^{-1}Z'e} - \sqrt{z_iA^{-1}z_i}\right)^2 \tag{1} \\ \geq & n^{-1}. \end{align}

In $$(1)$$, we used Cauchy-Schwarz inequality \begin{align} |z_i'A^{-1}Z'e|^2 \leq z_i'A^{-1}z_i \times e'ZA^{-1}Z'e. \end{align}

Proof of $$A$$ is invertible: Since $$P$$ is idempotent, to show $$A$$ is invertible, it suffices to show $$\operatorname{rank}(PZ) = p - 1$$, which (by rank-nullity theorem) is implied by $$PZx = 0$$ only has $$0$$ solution. If $$PZx = 0$$, then $$Zx \in \operatorname{Ker}(I_{(n)} - n^{-1}ee') = \operatorname{span}(e)$$. But since $$X$$ has full column rank, $$e$$ and $$Zx \in \operatorname{span}(Z)$$ are linearly independent, whence $$Zx = 0$$, which implies $$x = 0$$ due to $$\operatorname{rank}(Z) = p - 1$$. This completes the proof.

Here is another simpler (and perhaps more illuminating) proof that is based on QR decomposition of the design matrix $$X$$.

Suppose the QR decomposition of $$X$$ is $$X = QR$$, where $$Q \in \mathbb{R}^{n \times p}$$ is a matrix whose columns are orthogonal (so that $$Q'Q = I_{(p)}$$), $$R \in \mathbb{R}^{p \times p}$$ is an upper-triangular matrix. Since $$\operatorname{rank}(X) = p$$, $$\operatorname{rank}(R)$$ must be at least $$p$$, thus $$R$$ is invertible. Also note that since the first column of $$X$$ is $$e$$, the Gram-Schmidt procedure implies that the first column of $$Q$$ is $$\frac{1}{\sqrt{n}}e$$. Denote by $$e_i$$ the length-$$n$$ column vector of all zeros but the $$i$$-th position $$1$$, it then follows that \begin{align} & h_{ii} = e_i'X(X'X)^{-1}X'e_i = e_i'QR(R'Q'QR)^{-1}R'Q'e_i \\ =& e_i'QR(R'R)^{-1}R'Q'e_i = e_i'QRR^{-1}(R')^{-1}R'Q'e_i \\ =& e_i'QQ'e_i = \tilde{q}_i'\tilde{q}_i \\ =& n^{-1} + Q_{i2}^2 + \cdots + Q_{ip}^2 \geq n^{-1}, \end{align} where $$\tilde{q}_i' = \begin{bmatrix}\frac{1}{\sqrt{n}} & Q_{i2} & \cdots & Q_{ip}\end{bmatrix}$$ denotes the $$i$$-th row of $$Q$$. This completes the proof.

More details on QR decomposition: Suppose $$X = \begin{bmatrix} e & x_2 & \cdots & x_p \end{bmatrix}$$. By assumption, $$\{e, x_2, \ldots, x_p\}$$ are linearly independent, which allows us to apply the Gram-Schmidt procedure to obtain an orthonormal group $$\{q_1, q_2, \ldots, q_p\}$$ based on $$\{e, x_2, \ldots, x_p\}$$ as follows: \begin{align} & z_1 = e, \; q_1 = \frac{z_1}{\|z_1\|}, \tag{1} \\ & z_2 = x_2 - \frac{x_2'z_1}{z_1'z_1}z_1, \; q_2 = \frac{z_2}{\|z_2\|}, \\ & \cdots \cdots \cdots \\ & z_p = x_p - \frac{x_p'z_{n - 1}}{z_{n - 1}'z_{n - 1}}z_{n - 1} - \cdots - \frac{x_p'z_1}{z_1'z_1}z_1, \; q_p = \frac{z_p}{\|z_p\|}. \end{align} In matrical form, the above transformation can be recorded as \begin{align} X = \begin{bmatrix} e & x_2 & \cdots & x_p \end{bmatrix} &= \begin{bmatrix} q_1 & q_2 & \cdots & q_p \end{bmatrix} \begin{pmatrix} \|z_1\| & \frac{x_2'z_1}{\|z_1\|} & \cdots & \frac{x_p'z_1}{\|z_1\|} \\ & \|z_2\| & \cdots & \frac{x_p'z_2}{\|z_2\|} \\ & & \ddots & \vdots \\ & & & \|z_p\| \end{pmatrix}\\ &=: QR. \tag{2} \end{align} It is thus evident from $$(1)$$ and $$(2)$$ that $$q_1 = \frac{1}{\sqrt{n}}e$$.

$$H = H^\top H$$ implies $$h_{ii} = h_{ii}^2 + \sum_{j \neq i} h_{ij}^2$$, so $$h_{ii} \geq h_{ii}^2$$, hence $$0 \leq h_{ii} \leq 1$$.
Assuming presence of an intercept, we have $$h_{ii} = 1/n + D_i^2/\left(n-1\right) \geq 1/n$$, where $$D_i^2$$ is the squared Mahalanobis distance between the $$i$$-th row of a design matrix $$X$$ with 0-centered regressors and the origin.

• I do not understand why $h_{ii} = h_{ii}^2 + \sum_{j\ne i}h_{ij}^2$, can you clarify? May 11, 2023 at 0:24