11
$\begingroup$

I am having some problem trying to prove that the diagonal elements of the hat matrix $h_{ii}$ are between $1/n$ and $1$.

Suppose that $\mathrm{Range}(X_{n,k})=K $ the number of columns of our matrix of data with a constant.$\implies H_{k,k}$

$H=X(X' X)^{-1}X' \implies H'=H ;H^{2}=H $

If $y = \beta x + \epsilon \implies HY =\hat Y; (I-H)=\epsilon $

$\mathbf{H}=\begin{bmatrix}h_{11} &... &h_{1n}\\⋮ & ⋱ &⋮\\ h_{n1} & ... & h_{nn}\end{bmatrix}$

If $\boldsymbol 1=(1, \ldots, 1) \in X \implies 1H^2=1H=1 \implies \sum h_{i1}^2=\sum h_{i1}=1 , h_{ii}\leq 1$

So $∑h_{i1}^2\sum h_{i2}^2...\sum h_{in}^2=1 \implies h_{11}^2h_{22}^2...h_{nn}^2\leq 1 $

How to prove that $h_{ii} \geq(1/n)$?

(exercise 3.4 from Meyer "Classical and modern regression with applications") Let $h_{ii}$ be the ith diagonal of the Hat matrix H. (a) prove that for a multiple regression model with a constant term, $h_{ii} \geq(1/n)$ ; (b) show that $h_{ii}\leq 1$ (Hint: make the use of the fact that H is idempotent)

$\endgroup$
6
  • 4
    $\begingroup$ The result is not generally true: the diagonal elements can be less than $1/n$ when $X'X$ is not of full rank (and the generalized inverse is used). $\endgroup$
    – whuber
    Jun 17, 2013 at 14:33
  • 1
    $\begingroup$ $H$, as a projection matrix, is rarely of full rank. Even when $X'X$ is invertible, your conclusion is incorrect. Consider $X=(1,2)'$, where $X'X=(5)$, $(X'X)^{-1}=(1/5)$, and $H=X(X'X)^{-1}X'=\left( \begin{array}{cc} \frac{1}{5} & \frac{2}{5} \\ \frac{2}{5} & \frac{4}{5} \end{array} \right)$ has a diagonal entry less than $1/n=1/2$. $\endgroup$
    – whuber
    Jun 17, 2013 at 14:47
  • 2
    $\begingroup$ Please don't put your assumptions into comments: edit the question to include all the assumptions you wish to make. $\endgroup$
    – whuber
    Jun 17, 2013 at 14:52
  • 1
    $\begingroup$ $H$ will be of full rank only when $X$ is square: in that case you will no longer be doing least squares, but merely solving a completely determined set of linear equations. In general, the rank of $H$ does not exceed the number of columns of the design matrix $X$. In my counterexample, $X$ has one column and the rank of $H$ is one: as large as possible. $\endgroup$
    – whuber
    Jun 17, 2013 at 16:53
  • $\begingroup$ @whuber you are right my apollogies. When I said full range I mean that the range(X)= the number of regressor $\endgroup$
    – EAguirre
    Jun 17, 2013 at 17:33

6 Answers 6

9
$\begingroup$

This is several years later, but I found the notation very difficult in the asker's question and self-answer, so here's a cleaner solution.

We have $\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$ where $(1,...,1)^T$ is a column of $\mathbf{X}$. We want to show that the diagonals $h_{ii}$ of $\mathbf{H}$ have $h_{ii} \geq 1/n$.

Define $\mathbf{P} = \mathbf{H} - \mathbf{C}$, where

$$\mathbf{C} = \frac{1}{n}\begin{pmatrix}1 & \dots & 1 \\ \vdots & \ddots & \vdots \\ 1 & \dots & 1 \end{pmatrix}$$

the matrix consisting of only $1/n$. This is the projection matrix onto the space spanned by $(1, ..., 1)$. Then

$$\mathbf{P}^2 = \mathbf{H}^2 - \mathbf{H}\mathbf{C} - \mathbf{C}\mathbf{H} + \mathbf{C}^2 = \mathbf{H} - \mathbf{H}\mathbf{C} - \mathbf{C}\mathbf{H} + \mathbf{C}$$

However, $\mathbf{H}$ orthogonally projects onto $\text{Col}(\mathbf{X})$, and $\mathbf{C}$ orthogonally projects onto $\text{span}\{(1,...,1)\} \subset \text{Col}(\mathbf{X})$, so obviously $\mathbf{H}\mathbf{C} = \mathbf{C}$. Still intuitively, but less obviously, $\mathbf{C}\mathbf{H} = \mathbf{C}$. To see this, we can compute $\mathbf{C} = \mathbf{C}\big(\mathbf{H} + (\mathbf{I} - \mathbf{H})\big)$, and note that $\mathbf{C}(\mathbf{I} - \mathbf{H}) = 0$ because $\mathbf{I} - \mathbf{H}$ projects onto $\text{Col}(\mathbf{X})^\perp$.

Therefore we have $\mathbf{P}^2 = \mathbf{H} - \mathbf{C} = \mathbf{P}$. So $\mathbf{P}$ is also a projection matrix.

So $h_{ii} = p_{ii} + c_{ii} = p_{ii} + 1/n$. Since projection matrices are always positive semidefinite, the diagonals of $\mathbf{P}$ satisfy $p_{ii} \geq 0$. (In fact, you can show that since $\mathbf{P}$ is symmetric and idempotent, it satisfies $0 \leq p_{ii} \leq 1$.)

Then $h_{ii} \geq 1/n$ as needed.

$\endgroup$
5
  • $\begingroup$ Could you explain how it follows that "$P$ is also a projection matrix"? $\endgroup$
    – whuber
    Apr 24, 2019 at 2:44
  • $\begingroup$ Thanks, I glossed over that at first. Clarification now added. $\endgroup$
    – Drew N
    Apr 24, 2019 at 15:14
  • 1
    $\begingroup$ Thank you: I felt it was important to make that explicit connection with the assumption that the constant vector is in the span of the columns of $X.$ $\endgroup$
    – whuber
    Apr 24, 2019 at 16:26
  • $\begingroup$ What means of $\mathbf{Col(X)}$? $\endgroup$ Nov 11, 2021 at 9:28
  • $\begingroup$ @aminroshani It should be the space spanned by the column vectors of $\mathbf{X}$. $\endgroup$
    – Huihang
    Sep 15, 2022 at 13:05
4
$\begingroup$

For prove that $h_{ii} \geq (1/n)$, we can center $H_c=X(X_c' X_c)^{-1}X_c'$ , $\mathbf{H_c}=\begin{bmatrix}x_{11}-\bar x_1 &... &x_{1n}-\bar x_1 \\⋮ & ⋱ &⋮\\ x_{n1}-\bar x_n & ... & x_{nn}-\bar x_n\end{bmatrix}$

$y=\alpha1+ X_c'\beta +\epsilon⇒ \hat y=\hat \alpha1+ X_c'\hat\beta ⇒ \hat y=\bar y+ X_c'\hat\beta= \bar y+ X_c'(X_c' X_c)^{-1}X_c'y⇒ \hat y=[(1/n) 1'y]1+H_cy$ $=[1/n\begin{bmatrix}1&... &1\\⋮ & ⋱ &⋮\\1 & ... & 1\end{bmatrix}+H_c ] y=Hy $

Then $ H=1/n\begin{bmatrix}1&... &1\\⋮ & ⋱ &⋮\\1 & ... & 1\end{bmatrix}+H_c $$h_{ii} \geq (1/n)$ because $H_c$ is a positive definite matrix.

$\endgroup$
2
$\begingroup$

Here is another answer that that only uses the fact that all the eigenvalues of a symmetric idempotent matrix are at most 1, see one of the previous answers or prove it yourself, it's quite easy.

Let $H$ denote the hat matrix. The $i$th diagonal element of the hat matrix is given by

$$h_{ii} = \mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i,$$

where $\mathbf{e}_i^{t}$ is the vector whose $i$th element is 1 and the rest are 0s. Consider the quadratic form on the unit sphere given by

$$ f(\mathbf{x}) = \frac{\mathbf{x}^{t} \mathbf{H} \mathbf{x}}{\mathbf{x}^{t} \mathbf{x}}. $$

It is well known that the maximum of this expression is $\lambda_n$, the largest eigenvalue of the matrix $\mathbf{H}$. Returning to the diagonal elements of the hat matrix, one therefore has

$$h_{ii} = \mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i = \frac{\mathbf{e}_i^{t} \mathbf{H} \mathbf{e}_i}{\mathbf{e}_i^{t} \mathbf{e}_i} \underbrace{\mathbf{e}_i^{t} \mathbf{e}_i}_{ = 1} \leq \lambda_n \leq1 $$

and this gives us what we need.

$\endgroup$
1
  • 1
    $\begingroup$ How does this argument help prove $h_{ii} \geq \frac{1}{n}$? $\endgroup$
    – Zhanxiong
    Dec 30, 2022 at 1:57
0
$\begingroup$

Unlike the clever "centering" proof given by Drew N, the proof below is kind of brutal-force (but it has the advantage of giving even a sharper lower bound than $n^{-1}$, see $(1)$).

Partition the design matrix $X \in \mathbb{R}^{n \times p}$ to be $X = \begin{bmatrix} e & Z\end{bmatrix}$, where $e$ is the intercept term consisting of $n$ ones, $Z$ is the matrix consisting of $X$'s remaining $p - 1$ columns. By block matrix inversion formula, \begin{align} & (X'X)^{-1} = \begin{bmatrix} n & e'Z \\ Z'e & Z'Z \end{bmatrix}^{-1} \\ =& \begin{bmatrix} n^{-1} + n^{-2}e'Z(Z'PZ)^{-1}Z'e & -n^{-1}e'Z(Z'PZ)^{-1} \\ -n^{-1}(Z'PZ)^{-1}Z'e & (Z'PZ)^{-1} \end{bmatrix}, \end{align} where $P = I_{(n)} - n^{-1}ee'$ is idempotent, hence $A := Z'PZ$ is invertible (hence positive definite. The invertibility of $A$ is proved in detail at the end). It then follows by $h_{ii} = x_i'(X'X)^{-1}x_i$ that \begin{align} & h_{ii} = \begin{bmatrix} 1 & z_i' \end{bmatrix} \begin{bmatrix} n^{-1} + n^{-2}e'ZA^{-1}Z'e & -n^{-1}e'ZA^{-1} \\ -n^{-1}A^{-1}Z'e & A^{-1} \end{bmatrix} \begin{bmatrix} 1 \\ z_i \end{bmatrix} \\ =& \begin{bmatrix} n^{-1} + n^{-2}e'ZA^{-1}Z'e - n^{-1}z_i'A^{-1}Z'e & - n^{-1}e'ZA^{-1} + z_i'A^{-1} \end{bmatrix} \begin{bmatrix} 1 \\ z_i \end{bmatrix} \\ =& n^{-1} + n^{-2}e'ZA^{-1}Z'e - n^{-1}z_i'A^{-1}Z'e - n^{-1}e'ZA^{-1}z_i + z_i'A^{-1}z_i \\ \geq & n^{-1} + \left(n^{-1}\sqrt{e'ZA^{-1}Z'e} - \sqrt{z_iA^{-1}z_i}\right)^2 \tag{1} \\ \geq & n^{-1}. \end{align}

In $(1)$, we used Cauchy-Schwarz inequality \begin{align} |z_i'A^{-1}Z'e|^2 \leq z_i'A^{-1}z_i \times e'ZA^{-1}Z'e. \end{align}


Proof of $A$ is invertible: Since $P$ is idempotent, to show $A$ is invertible, it suffices to show $\operatorname{rank}(PZ) = p - 1$, which (by rank-nullity theorem) is implied by $PZx = 0$ only has $0$ solution. If $PZx = 0$, then $Zx \in \operatorname{Ker}(I_{(n)} - n^{-1}ee') = \operatorname{span}(e)$. But since $X$ has full column rank, $e$ and $Zx \in \operatorname{span}(Z)$ are linearly independent, whence $Zx = 0$, which implies $x = 0$ due to $\operatorname{rank}(Z) = p - 1$. This completes the proof.

$\endgroup$
0
$\begingroup$

Here is another simpler (and perhaps more illuminating) proof that is based on QR decomposition of the design matrix $X$.

Suppose the QR decomposition of $X$ is $X = QR$, where $Q \in \mathbb{R}^{n \times p}$ is a matrix whose columns are orthogonal (so that $Q'Q = I_{(p)}$), $R \in \mathbb{R}^{p \times p}$ is an upper-triangular matrix. Since $\operatorname{rank}(X) = p$, $\operatorname{rank}(R)$ must be at least $p$, thus $R$ is invertible. Also note that since the first column of $X$ is $e$, the Gram-Schmidt procedure implies that the first column of $Q$ is $\frac{1}{\sqrt{n}}e$. Denote by $e_i$ the length-$n$ column vector of all zeros but the $i$-th position $1$, it then follows that \begin{align} & h_{ii} = e_i'X(X'X)^{-1}X'e_i = e_i'QR(R'Q'QR)^{-1}R'Q'e_i \\ =& e_i'QR(R'R)^{-1}R'Q'e_i = e_i'QRR^{-1}(R')^{-1}R'Q'e_i \\ =& e_i'QQ'e_i = \tilde{q}_i'\tilde{q}_i \\ =& n^{-1} + Q_{i2}^2 + \cdots + Q_{ip}^2 \geq n^{-1}, \end{align} where $\tilde{q}_i' = \begin{bmatrix}\frac{1}{\sqrt{n}} & Q_{i2} & \cdots & Q_{ip}\end{bmatrix}$ denotes the $i$-th row of $Q$. This completes the proof.


More details on QR decomposition: Suppose $X = \begin{bmatrix} e & x_2 & \cdots & x_p \end{bmatrix}$. By assumption, $\{e, x_2, \ldots, x_p\}$ are linearly independent, which allows us to apply the Gram-Schmidt procedure to obtain an orthonormal group $\{q_1, q_2, \ldots, q_p\}$ based on $\{e, x_2, \ldots, x_p\}$ as follows: \begin{align} & z_1 = e, \; q_1 = \frac{z_1}{\|z_1\|}, \tag{1} \\ & z_2 = x_2 - \frac{x_2'z_1}{z_1'z_1}z_1, \; q_2 = \frac{z_2}{\|z_2\|}, \\ & \cdots \cdots \cdots \\ & z_p = x_p - \frac{x_p'z_{n - 1}}{z_{n - 1}'z_{n - 1}}z_{n - 1} - \cdots - \frac{x_p'z_1}{z_1'z_1}z_1, \; q_p = \frac{z_p}{\|z_p\|}. \end{align} In matrical form, the above transformation can be recorded as \begin{align} X = \begin{bmatrix} e & x_2 & \cdots & x_p \end{bmatrix} &= \begin{bmatrix} q_1 & q_2 & \cdots & q_p \end{bmatrix} \begin{pmatrix} \|z_1\| & \frac{x_2'z_1}{\|z_1\|} & \cdots & \frac{x_p'z_1}{\|z_1\|} \\ & \|z_2\| & \cdots & \frac{x_p'z_2}{\|z_2\|} \\ & & \ddots & \vdots \\ & & & \|z_p\| \end{pmatrix}\\ &=: QR. \tag{2} \end{align} It is thus evident from $(1)$ and $(2)$ that $q_1 = \frac{1}{\sqrt{n}}e$.

$\endgroup$
0
$\begingroup$

$H = H^\top H$ implies $h_{ii} = h_{ii}^2 + \sum_{j \neq i} h_{ij}^2$, so $h_{ii} \geq h_{ii}^2$, hence $0 \leq h_{ii} \leq 1$.
Assuming presence of an intercept, we have $h_{ii} = 1/n + D_i^2/\left(n-1\right) \geq 1/n$, where $D_i^2$ is the squared Mahalanobis distance between the $i$-th row of a design matrix $X$ with 0-centered regressors and the origin.

$\endgroup$
1
  • $\begingroup$ I do not understand why $h_{ii} = h_{ii}^2 + \sum_{j\ne i}h_{ij}^2$, can you clarify? $\endgroup$ May 11, 2023 at 0:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.