# Diagonal elements of the projection matrix

I am having some problem trying to prove that the diagonal elements of the hat matrix $h_{ii}$ are between $1/n$ and $1$.

Suppose that $Range(X_{n,k})=K$ the number of columns of our matrix of data with a constant.⇒$H_{k,k}$

$H=X(X' X)^{-1}X' ⇒ H'=H ;H^{2}=H$

If $y = \beta x + \epsilon ⇒ HY =\hat Y; (I-H)=\epsilon$

$\mathbf{H}=\begin{bmatrix}h_{11} &... &h_{1n}\\⋮ & ⋱ &⋮\\ h_{n1} & ... & h_{nn}\end{bmatrix}$

If $\boldsymbol 1=(1, \ldots, 1) ∈ X ⇒ 1H^2=1H=1 ⇒ ∑ h_{i1}^2=∑ h_{i1}=1 , h_{ii}≤1$

So $∑h_{i1}^2∑h_{i2}^2...∑h_{in}^2=1 ⇒h_{11}^2h_{22}^2...h_{nn}^2<=1$

How to prove that $h_{ii} >=(1/n)$?

(exercise 3.4 from Meyer "Classical and modern regression with applications") Let $h_{ii}$ be the ith diagonal of the Hat matrix H. (a) prove that for a multiple regression model with a constant term, $h_{ii} >=(1/n)$ ; (b) show that $h_{ii}≤1$ (Hint: make the use of the fact that H is idempotent)

• (1) You are missing a transpose in your definition of $H$. (2) The result is not generally true: the diagonal elements can be less than $1/n$ when $X'X$ is not of full rank (and the generalized inverse is used). – whuber Jun 17 '13 at 14:33
• $H$, as a projection matrix, is rarely of full rank. Even when $X'X$ is invertible, your conclusion is incorrect. Consider $X=(1,2)'$, where $X'X=(5)$, $(X'X)^{-1}=(1/5)$, and $H=X(X'X)^{-1}X'=\left( \begin{array}{cc} \frac{1}{5} & \frac{2}{5} \\ \frac{2}{5} & \frac{4}{5} \end{array} \right)$ has a diagonal entry less than $1/n=1/2$. – whuber Jun 17 '13 at 14:47
• Please don't put your assumptions into comments: edit the question to include all the assumptions you wish to make. – whuber Jun 17 '13 at 14:52
• $H$ will be of full rank only when $X$ is square: in that case you will no longer be doing least squares, but merely solving a completely determined set of linear equations. In general, the rank of $H$ does not exceed the number of columns of the design matrix $X$. In my counterexample, $X$ has one column and the rank of $H$ is one: as large as possible. – whuber Jun 17 '13 at 16:53
• @whuber you are right my apollogies. When I said full range I mean that the range(X)= the number of regressor – EAguirre Jun 17 '13 at 17:33

This is several years later, but I found the notation very difficult in the asker's question and self-answer, so here's a cleaner solution.

We have $$\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$$ where $$(1,...,1)^T$$ is a column of $$\mathbf{X}$$. We want to show that the diagonals $$h_{ii}$$ of $$\mathbf{H}$$ have $$h_{ii} \geq 1/n$$.

Define $$\mathbf{P} = \mathbf{H} - \mathbf{C}$$, where

$$\mathbf{C} = \frac{1}{n}\begin{pmatrix}1 & \dots & 1 \\ \vdots & \ddots & \vdots \\ 1 & \dots & 1 \end{pmatrix}$$

the matrix consisting of only $$1/n$$. This is the projection matrix onto the space spanned by $$(1, ..., 1)$$. Then

$$\mathbf{P}^2 = \mathbf{H}^2 - \mathbf{H}\mathbf{C} - \mathbf{C}\mathbf{H} + \mathbf{C}^2 = \mathbf{H} - \mathbf{H}\mathbf{C} - \mathbf{C}\mathbf{H} + \mathbf{C}$$

However, $$\mathbf{H}$$ orthogonally projects onto $$\text{Col}(\mathbf{X})$$, and $$\mathbf{C}$$ orthogonally projects onto $$\text{span}\{(1,...,1)\} \subset \text{Col}(\mathbf{X})$$, so obviously $$\mathbf{H}\mathbf{C} = \mathbf{C}$$. Still intuitively, but less obviously, $$\mathbf{C}\mathbf{H} = \mathbf{C}$$. To see this, we can compute $$\mathbf{C} = \mathbf{C}\big(\mathbf{H} + (\mathbf{I} - \mathbf{H})\big)$$, and note that $$\mathbf{C}(\mathbf{I} - \mathbf{H}) = 0$$ because $$\mathbf{I} - \mathbf{H}$$ projects onto $$\text{Col}(\mathbf{X})^\perp$$.

Therefore we have $$\mathbf{P}^2 = \mathbf{H} - \mathbf{C} = \mathbf{P}$$. So $$\mathbf{P}$$ is also a projection matrix.

So $$h_{ii} = p_{ii} + c_{ii} = p_{ii} + 1/n$$. Since projection matrices are always positive semidefinite, the diagonals of $$\mathbf{P}$$ satisfy $$p_{ii} \geq 0$$. (In fact, you can show that since $$\mathbf{P}$$ is symmetric and idempotent, it satisfies $$0 \leq p_{ii} \leq 1$$.)

Then $$h_{ii} \geq 1/n$$ as needed.

• Could you explain how it follows that "$P$ is also a projection matrix"? – whuber Apr 24 at 2:44
• Thanks, I glossed over that at first. Clarification now added. – Drew N Apr 24 at 15:14
• Thank you: I felt it was important to make that explicit connection with the assumption that the constant vector is in the span of the columns of $X.$ – whuber Apr 24 at 16:26

For prove that $h_{ii} >=(1/n)$, we can center $H_c=X(X_c' X_c)^{-1}X_c'$ , $\mathbf{H_c}=\begin{bmatrix}x_{11}-\bar x_1 &... &x_{1n}-\bar x_1 \\⋮ & ⋱ &⋮\\ x_{n1}-\bar x_n & ... & x_{nn}-\bar x_n\end{bmatrix}$

$y=\alpha1+ X_c'\beta +\epsilon⇒ \hat y=\hat \alpha1+ X_c'\hat\beta ⇒ \hat y=\bar y+ X_c'\hat\beta= \bar y+ X_c'(X_c' X_c)^{-1}X_c'y⇒ \hat y=[(1/n) 1'y]1+H_cy$ $=[1/n\begin{bmatrix}1&... &1\\⋮ & ⋱ &⋮\\1 & ... & 1\end{bmatrix}+H_c ] y=Hy$

Then $H=1/n\begin{bmatrix}1&... &1\\⋮ & ⋱ &⋮\\1 & ... & 1\end{bmatrix}+H_c$⇒ $h_{ii} >=(1/n)$ because $H_c$ is a positive definite matrix

Here is another answer that that only uses the fact that all the eigenvalues of a symmetric idempotent matrix are at most 1, see one of the previous answers or prove it yourself, it's quite easy.

Let $$H$$ denote the hat matrix. The $$i$$th diagonal element of the hat matrix is given by

$$h_{ii} = \mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i,$$

where $$\mathbf{e}_i^{t}$$ is the vector whose $$i$$th element is 1 and the rest are 0s. Consider the quadratic form on the unit sphere given by

$$f(\mathbf{x}) = \frac{\mathbf{x}^{t} \mathbf{H} \mathbf{x}}{\mathbf{x}^{t} \mathbf{x}}.$$

It is well known that the maximum of this expression is $$\lambda_n$$, the largest eigenvalue of the matrix $$\mathbf{H}$$. Returning to the diagonal elements of the hat matrix, one therefore has

$$h_{ii} = \mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i = \frac{\mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i}{\mathbf{e}_i^{t} \mathbf{e}_i} \underbrace{\mathbf{e}_i^{t} \mathbf{e}_i}_{ = 1} \leq \lambda_n \leq1$$

and this gives us what we need.