In general linear model $$Y=X\beta +\epsilon $$

the LSE for $\beta$ is $$\hat \beta=(X^TX)^{-1}X^TY$$ and so $$\hat Y=X\hat \beta=X(X^TX)^{-1}X^TY=HY$$ where $H=X(X^TX)^{-1}X^T$.

Then the residual is $$E=Y-\hat Y=(I_n-H)Y$$

Substitute $Y=X\beta +\epsilon$ we get $$E=(I_n-H)X\beta+(I_n-H)\epsilon=X\beta-X(X^TX)^{-1}X^TX\beta+(I_n-H)\epsilon=(I_n-H)\epsilon$$

Observing the last two expressions, we get $$E=(I_n-H)Y=(I_n-H)\epsilon$$ Then $Y=\epsilon$? Is this correct? What does this mean? Could anyone explain how this comes?


No it's not correct. Let $A$ be an $n \times n$ matrix and $w, v$ two $n-$vectors. Think about under what conditions $Av = Aw$ implies $v = w$ and check whether those conditions hold for your case.

Here are two ways to prove non-invertibility. They may require you to read about projections and projection matrices.

To show that $Q:=I_n - H$ is not invertible in general, you may note that if $Q$ is invertible, then its columns span $\mathbb R^n$, as any linearly independent set of $n-$vectors does. This means the orthogonal complement of the column space of $Q$ is the zero vector. But the orthogonal complement of the column space of $Q$ is the column space of $X$. Thus, $Q$ is invertible iff $X = 0$.

Another way of arguing that $Q$ is not invertible is to prove that projection matrices have eigenvalues that are either $1$ or $0$. Thus, if $Q$ is invertible all its eigenvalues are $1$, but this implies all the eigenvalues of $H$ are zero, which again implies that $X = 0$.

  • $\begingroup$ Sorry I haven't learned about projection matrix so I didn't quite understand your second paragraph. But I guess do you mean $I_n-H$ must be invertible which might not be the case? $\endgroup$ – J.Y May 14 '16 at 15:32
  • $\begingroup$ Ok, that's fine. The second paragraph is useful if you want to understand what's going on (well, at least I think so), but you don't need it to realize why the suggested implication doesn't hold. You are right, the rank of $I_n - H$ is the issue. $\endgroup$ – ekvall May 14 '16 at 15:34
  • $\begingroup$ Thank you. Is there any property necessary for the data so that $I_n-H$ is invertible? Or is that true that in general it is not invertible? $\endgroup$ – J.Y May 14 '16 at 15:41
  • $\begingroup$ @J.Y I changed the answer somewhat. In general $Q$ is not invertible. $\endgroup$ – ekvall May 14 '16 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.