2
$\begingroup$

This is in multiple linear regression. Given $m, n \in \mathbb{N}$ and matrices $X \in \mathbb{R}^{m \times (n+1)} (m > n + 1), H = X(X'X)^{-1}X' \in \mathbb{R}^{m\times m}$, the hat matrix, $, I = I_m$ and $J \in \mathbb{R}^{m \times m}$, a matrix of 1's, how does one show that

$J(I-H) = 0$?

I am actually trying to show

$(I-H)(H-\frac{1}{n}J) = 0$

which reduces to $J(I-H) = 0$ by noting that $HH =H$.

Apparently, this might have something to do with $J(y - X \hat{\beta}) = 0 \ \forall y \in \mathbb{R}^{1 \times (m+1)}$ where $\hat{\beta} = (X'X)^{-1}X'y $?

Is the statement true? If so why? If not, why not, and how then to prove $J(I-H) = 0$ or $(I-H)(H-\frac{1}{n}J) = 0$? I think there's some property of MLR needed.

Is there a way to do it without statistics? Like $J(I-H) = 0$ or $(I-H)(H-\frac{1}{n}J) = 0$ from just the properties of the matrices? Must one really use some properties of MLR? What are some possible counterexamples?

$\endgroup$
0

4 Answers 4

4
$\begingroup$

The "hat" matrix is a projection matrix onto the column space of $X$, $C(X)$. For any $x \in C(X)$, $Hx=x$ (see properties of projection matrix). Therefore $JH=J$ as long as you have an intercept (column of ones in your design matrix).

EDIT to address comment:

The matrix of all ones, $J$, can be written as $$J = \begin{pmatrix} \mathbf{1}^\intercal \\ \vdots \\\mathbf{1}^\intercal\end{pmatrix}$$ where $\mathbf{1}$ is a vector of ones. Then we have $$JH = \begin{pmatrix} \mathbf{1}^\intercal H \\ \vdots \\ \mathbf{1}^\intercal H \end{pmatrix} = \begin{pmatrix} \mathbf{1}^\intercal \\ \vdots \\ \mathbf{1}^\intercal\end{pmatrix} = J$$

if column of ones is in $\mathbf{X}$ then $H\mathbf{1} = \mathbf{1}$ and taking transpose of both sides we have $\mathbf{1}^\intercal H = \mathbf{1}^\intercal$ because $H$ is symmetric.

$\endgroup$
8
  • $\begingroup$ There indeed is a column of ones. How exactly does it mean that JH = J? What I got so far: Column space means set of all possible linear combinations of X. It is a set of vectors. How can J be there if it is a matrix? en.wikipedia.org/wiki/Column_space#Definition $\endgroup$
    – BCLC
    May 14, 2015 at 15:28
  • $\begingroup$ I hope my edit answers your question. :) $\endgroup$
    – bdeonovic
    May 14, 2015 at 16:46
  • $\begingroup$ bdeonovic, why is it that 1TH = 1T? $\endgroup$
    – BCLC
    May 14, 2015 at 16:47
  • $\begingroup$ I had the property on projection matrices a bit wrong, its been corrected and added another edit to answer your second comment. $\endgroup$
    – bdeonovic
    May 14, 2015 at 16:58
  • $\begingroup$ Ugh, how do you know H1 = 1? $\endgroup$
    – BCLC
    May 14, 2015 at 17:16
2
$\begingroup$

This can be proved with just the properties of the matrices, since the MLR is already baked into $H$.

If $H$ is defined as $H:=X(X'X)^{-1}X'$ then clearly $HX=X$. Note that $J=11^T$, where $1$ is a column of ones. Suppose that $1$ lives in the $k$th column of $X$, which I'll write as $(X)_{\ast k}=1$.

Since the $k$th column of $HX$ equals $H$ times the $k$th column of $X$, we can show $H1=1$: $$H1=H(X)_{\ast k}=(HX)_{\ast k}=(X)_{\ast k}=1$$ and therefore $$HJ=H11^T=(H1)1^T=11^T=J.$$

$\endgroup$
3
  • $\begingroup$ forgot this already, but sounds like an answer 6 years ago me was looking for. thanks! $\endgroup$
    – BCLC
    Sep 9, 2021 at 21:46
  • 1
    $\begingroup$ I like this answer for using simply the property of the matrix $HX = X(X^tX)^-1(X^tX)= X$. But for $\mathbf{1}$ to be an eigenvector of $H$ it doesn't necessarily need to be a column of $X$ (it would be the easiest case to show it) and it just needs to be in the column space. $\endgroup$ Sep 25, 2021 at 18:51
  • 1
    $\begingroup$ @SextusEmpiricus Indeed! If $1$ is in the column space of $X$ then there is a vector $v$ of coefficients such that $1=Xv$, and therefore $H1=H(Xv)=(HX)v=Xv=1$. I like this generalization better than the assumption that $1$ is a column in $X$. $\endgroup$
    – grand_chat
    Sep 25, 2021 at 19:17
1
$\begingroup$

Going for the shortest answer here: just look at the first row of $\mathbf{X}^\intercal \mathbf{H} = \mathbf{X}^\intercal$.

$\endgroup$
1
  • 1
    $\begingroup$ forgot this already, but sounds like an answer 6-8 years ago me was looking for. thanks! $\endgroup$
    – BCLC
    Sep 25, 2021 at 20:05
1
$\begingroup$

$J(H-I) = 0$ is another way to say $JH = J$.

If for a vector of only ones, let's write it as $\mathbf{1}$, you get $\mathbf{1}H = \mathbf{1}$ then it also works for $J$ as well.

This property $\mathbf{1}H = \mathbf{1}$ is actually only true when the matrix $X$ contains the vector $\mathbf{1}$ in it's column space, for instance when it contains an intercept term (but an intercept is not necessary).

Counter example. Let $X = \lbrace -1, 0, -1\rbrace$ then $$H = \begin{bmatrix} 0.5 & 0 & -0.5 \\ 0 & 0 & 0 \\ -0.5 & 0 & 0.5 \end{bmatrix}$$

and $\mathbf{1} H = \lbrace 0,0, 0 \rbrace \neq \mathbf{1}$

$\endgroup$
1
  • $\begingroup$ forgot this already, but sounds like an answer 6-8 years ago me was looking for. thanks! $\endgroup$
    – BCLC
    Sep 25, 2021 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.