In linear algebra class I learned, that $$\begin{equation*} \hat{Y} = X \hat{\beta} = X\,\left(X^\prime X \right)^{-1} \, X^\prime Y = P\,Y \end{equation*}$$

, where \begin{equation*} P \equiv X\,\left(X^\prime X \right)^{-1} \, X^\prime \end{equation*} is a projection matrix.

But in linear model course a projection matrix is defined in another way:

\begin{equation*} P_2 \equiv I_n- X\,\left(X^\prime X \right)^{-1} \, X^\prime \end{equation*}

Why is so? Why in regression analysis we need projection matrix $P_2$ instead of $P$?


I see more often the notation

\begin{equation*} M \equiv I_n- X\,\left(X^\prime X \right)^{-1} \, X^\prime \end{equation*}

where $M$ is called the "annihilator" matrix, (because $MX = 0$) or "residual maker" matrix because $MY = \hat u$.

We do call $P$ the orthogonal projection matrix, and here too it holds that $ \hat Y = PY$

| cite | improve this answer | |

A square matrix is a projection matrix if and only it is equal to its square.

You know that $P$ is a projection matrix, therefore (multiply it out), $P^2 = P$. Therefore, $P_2^2 = P_2$. Therefore $P_2$ is a projection matrix.(as is $P$).

Exercise for you: What is the relation between the subspaces to which $P$ and $P_2$ project?

| cite | improve this answer | |
  • $\begingroup$ They are orthogonal to each other. I think you incorrectly understood my question(sorry for that). I know, that $P_2$ is projection matrix. But the question is: why do we use $P_2$ in regression analysis(in my linear model class we are using only $P_2$, instead of $P$? Which purposes have $P$ and $P_2$? $\endgroup$ – Daniel Yefimov Nov 25 '16 at 18:52
  • 1
    $\begingroup$ Read pp. 5-7 as precursor, then pp. 8-10 of stat.cmu.edu/~ryantibs/datamining/lectures/13-reg1.pdf . $\endgroup$ – Mark L. Stone Nov 25 '16 at 19:02
  • $\begingroup$ is that another way of stating idempotency or is there an important reason why you define a projection matrix as equal to its own square, rather than saying: A projection matrix is idempotent? $\endgroup$ – Beyer Nov 25 '16 at 20:39
  • $\begingroup$ Idempotence of a matrix is a fancy way of saying that it equals its own square. I decided not to use terminology (a "big" word) not everyone would know - that was a judgment call - I'm just a working class Joe at heart. $\endgroup$ – Mark L. Stone Nov 25 '16 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.