I am having some problem trying to prove that the diagonal elements of the hat matrix $h_{ii}$ are between $1/n$ and $1$.

Suppose that $Range(X_{n,k})=K $ the number of columns of our matrix of data with a constant.⇒$H_{k,k}$

$H=X(X' X)^{-1}X' ⇒ H'=H ;H^{2}=H $

If $y = \beta x + \epsilon ⇒ HY =\hat Y; (I-H)=\epsilon $

$\mathbf{H}=\begin{bmatrix}h_{11} &... &h_{1n}\\⋮ & ⋱ &⋮\\ h_{n1} & ... & h_{nn}\end{bmatrix}$

If $\boldsymbol 1=(1, \ldots, 1) ∈ X ⇒ 1H^2=1H=1 ⇒ ∑ h_{i1}^2=∑ h_{i1}=1 , h_{ii}≤1$

So $∑h_{i1}^2∑h_{i2}^2...∑h_{in}^2=1 ⇒h_{11}^2h_{22}^2...h_{nn}^2<=1 $

How to prove that $h_{ii} >=(1/n)$?

(exercise 3.4 from Meyer "Classical and modern regression with applications") Let $h_{ii}$ be the ith diagonal of the Hat matrix H. (a) prove that for a multiple regression model with a constant term, $h_{ii} >=(1/n)$ ; (b) show that $h_{ii}≤1$ (Hint: make the use of the fact that H is idempotent)

  • 3
    $\begingroup$ (1) You are missing a transpose in your definition of $H$. (2) The result is not generally true: the diagonal elements can be less than $1/n$ when $X'X$ is not of full rank (and the generalized inverse is used). $\endgroup$ – whuber Jun 17 '13 at 14:33
  • 1
    $\begingroup$ $H$, as a projection matrix, is rarely of full rank. Even when $X'X$ is invertible, your conclusion is incorrect. Consider $X=(1,2)'$, where $X'X=(5)$, $(X'X)^{-1}=(1/5)$, and $H=X(X'X)^{-1}X'=\left( \begin{array}{cc} \frac{1}{5} & \frac{2}{5} \\ \frac{2}{5} & \frac{4}{5} \end{array} \right)$ has a diagonal entry less than $1/n=1/2$. $\endgroup$ – whuber Jun 17 '13 at 14:47
  • 2
    $\begingroup$ Please don't put your assumptions into comments: edit the question to include all the assumptions you wish to make. $\endgroup$ – whuber Jun 17 '13 at 14:52
  • 1
    $\begingroup$ $H$ will be of full rank only when $X$ is square: in that case you will no longer be doing least squares, but merely solving a completely determined set of linear equations. In general, the rank of $H$ does not exceed the number of columns of the design matrix $X$. In my counterexample, $X$ has one column and the rank of $H$ is one: as large as possible. $\endgroup$ – whuber Jun 17 '13 at 16:53
  • $\begingroup$ @whuber you are right my apollogies. When I said full range I mean that the range(X)= the number of regressor $\endgroup$ – EAguirre Jun 17 '13 at 17:33

This is several years later, but I found the notation very difficult in the asker's question and self-answer, so here's a cleaner solution.

We have $\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$ where $(1,...,1)^T$ is a column of $\mathbf{X}$. We want to show that the diagonals $h_{ii}$ of $\mathbf{H}$ have $h_{ii} \geq 1/n$.

Define $\mathbf{P} = \mathbf{H} - \mathbf{C}$, where

$$\mathbf{C} = \frac{1}{n}\begin{pmatrix}1 & \dots & 1 \\ \vdots & \ddots & \vdots \\ 1 & \dots & 1 \end{pmatrix}$$

the matrix consisting of only $1/n$. This is the projection matrix onto the space spanned by $(1, ..., 1)$. Then

$$\mathbf{P}^2 = \mathbf{H}^2 - \mathbf{H}\mathbf{C} - \mathbf{C}\mathbf{H} + \mathbf{C}^2 = \mathbf{H} - \mathbf{H}\mathbf{C} - \mathbf{C}\mathbf{H} + \mathbf{C}$$

However, $\mathbf{H}$ orthogonally projects onto $\text{Col}(\mathbf{X})$, and $\mathbf{C}$ orthogonally projects onto $\text{span}\{(1,...,1)\} \subset \text{Col}(\mathbf{X})$, so obviously $\mathbf{H}\mathbf{C} = \mathbf{C}$. Still intuitively, but less obviously, $\mathbf{C}\mathbf{H} = \mathbf{C}$. To see this, we can compute $\mathbf{C} = \mathbf{C}\big(\mathbf{H} + (\mathbf{I} - \mathbf{H})\big)$, and note that $\mathbf{C}(\mathbf{I} - \mathbf{H}) = 0$ because $\mathbf{I} - \mathbf{H}$ projects onto $\text{Col}(\mathbf{X})^\perp$.

Therefore we have $\mathbf{P}^2 = \mathbf{H} - \mathbf{C} = \mathbf{P}$. So $\mathbf{P}$ is also a projection matrix.

So $h_{ii} = p_{ii} + c_{ii} = p_{ii} + 1/n$. Since projection matrices are always positive semidefinite, the diagonals of $\mathbf{P}$ satisfy $p_{ii} \geq 0$. (In fact, you can show that since $\mathbf{P}$ is symmetric and idempotent, it satisfies $0 \leq p_{ii} \leq 1$.)

Then $h_{ii} \geq 1/n$ as needed.

  • $\begingroup$ Could you explain how it follows that "$P$ is also a projection matrix"? $\endgroup$ – whuber Apr 24 at 2:44
  • $\begingroup$ Thanks, I glossed over that at first. Clarification now added. $\endgroup$ – Drew N Apr 24 at 15:14
  • 1
    $\begingroup$ Thank you: I felt it was important to make that explicit connection with the assumption that the constant vector is in the span of the columns of $X.$ $\endgroup$ – whuber Apr 24 at 16:26

For prove that $h_{ii} >=(1/n)$, we can center $H_c=X(X_c' X_c)^{-1}X_c'$ , $\mathbf{H_c}=\begin{bmatrix}x_{11}-\bar x_1 &... &x_{1n}-\bar x_1 \\⋮ & ⋱ &⋮\\ x_{n1}-\bar x_n & ... & x_{nn}-\bar x_n\end{bmatrix}$

$y=\alpha1+ X_c'\beta +\epsilon⇒ \hat y=\hat \alpha1+ X_c'\hat\beta ⇒ \hat y=\bar y+ X_c'\hat\beta= \bar y+ X_c'(X_c' X_c)^{-1}X_c'y⇒ \hat y=[(1/n) 1'y]1+H_cy$ $=[1/n\begin{bmatrix}1&... &1\\⋮ & ⋱ &⋮\\1 & ... & 1\end{bmatrix}+H_c ] y=Hy $

Then $ H=1/n\begin{bmatrix}1&... &1\\⋮ & ⋱ &⋮\\1 & ... & 1\end{bmatrix}+H_c $⇒ $h_{ii} >=(1/n)$ because $H_c$ is a positive definite matrix


Here is another answer that that only uses the fact that all the eigenvalues of a symmetric idempotent matrix are at most 1, see one of the previous answers or prove it yourself, it's quite easy.

Let $H$ denote the hat matrix. The $i$th diagonal element of the hat matrix is given by

$$h_{ii} = \mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i,$$

where $\mathbf{e}_i^{t}$ is the vector whose $i$th element is 1 and the rest are 0s. Consider the quadratic form on the unit sphere given by

$$ f(\mathbf{x}) = \frac{\mathbf{x}^{t} \mathbf{H} \mathbf{x}}{\mathbf{x}^{t} \mathbf{x}}. $$

It is well known that the maximum of this expression is $\lambda_n$, the largest eigenvalue of the matrix $\mathbf{H}$. Returning to the diagonal elements of the hat matrix, one therefore has

$$h_{ii} = \mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i = \frac{\mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i}{\mathbf{e}_i^{t} \mathbf{e}_i} \underbrace{\mathbf{e}_i^{t} \mathbf{e}_i}_{ = 1} \leq \lambda_n \leq1 $$

and this gives us what we need.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.