$Y=\epsilon$ in GLM?

In general linear model $$Y=X\beta +\epsilon$$

the LSE for $\beta$ is $$\hat \beta=(X^TX)^{-1}X^TY$$ and so $$\hat Y=X\hat \beta=X(X^TX)^{-1}X^TY=HY$$ where $H=X(X^TX)^{-1}X^T$.

Then the residual is $$E=Y-\hat Y=(I_n-H)Y$$

Substitute $Y=X\beta +\epsilon$ we get $$E=(I_n-H)X\beta+(I_n-H)\epsilon=X\beta-X(X^TX)^{-1}X^TX\beta+(I_n-H)\epsilon=(I_n-H)\epsilon$$

Observing the last two expressions, we get $$E=(I_n-H)Y=(I_n-H)\epsilon$$ Then $Y=\epsilon$? Is this correct? What does this mean? Could anyone explain how this comes?

1 Answer

No it's not correct. Let $A$ be an $n \times n$ matrix and $w, v$ two $n-$vectors. Think about under what conditions $Av = Aw$ implies $v = w$ and check whether those conditions hold for your case.

Here are two ways to prove non-invertibility. They may require you to read about projections and projection matrices.

To show that $Q:=I_n - H$ is not invertible in general, you may note that if $Q$ is invertible, then its columns span $\mathbb R^n$, as any linearly independent set of $n-$vectors does. This means the orthogonal complement of the column space of $Q$ is the zero vector. But the orthogonal complement of the column space of $Q$ is the column space of $X$. Thus, $Q$ is invertible iff $X = 0$.

Another way of arguing that $Q$ is not invertible is to prove that projection matrices have eigenvalues that are either $1$ or $0$. Thus, if $Q$ is invertible all its eigenvalues are $1$, but this implies all the eigenvalues of $H$ are zero, which again implies that $X = 0$.

• Sorry I haven't learned about projection matrix so I didn't quite understand your second paragraph. But I guess do you mean $I_n-H$ must be invertible which might not be the case? – J.Y May 14 '16 at 15:32
• Ok, that's fine. The second paragraph is useful if you want to understand what's going on (well, at least I think so), but you don't need it to realize why the suggested implication doesn't hold. You are right, the rank of $I_n - H$ is the issue. – ekvall May 14 '16 at 15:34
• Thank you. Is there any property necessary for the data so that $I_n-H$ is invertible? Or is that true that in general it is not invertible? – J.Y May 14 '16 at 15:41
• @J.Y I changed the answer somewhat. In general $Q$ is not invertible. – ekvall May 14 '16 at 16:04