# Projection Matrix in linear regression(and difference between Projection Matrix in linear Algebra)

In linear algebra class I learned, that $$\begin{equation*} \hat{Y} = X \hat{\beta} = X\,\left(X^\prime X \right)^{-1} \, X^\prime Y = P\,Y \end{equation*}$$

, where \begin{equation*} P \equiv X\,\left(X^\prime X \right)^{-1} \, X^\prime \end{equation*} is a projection matrix.

But in linear model course a projection matrix is defined in another way:

\begin{equation*} P_2 \equiv I_n- X\,\left(X^\prime X \right)^{-1} \, X^\prime \end{equation*}

Why is so? Why in regression analysis we need projection matrix $P_2$ instead of $P$?

I see more often the notation

\begin{equation*} M \equiv I_n- X\,\left(X^\prime X \right)^{-1} \, X^\prime \end{equation*}

where $M$ is called the "annihilator" matrix, (because $MX = 0$) or "residual maker" matrix because $MY = \hat u$.

We do call $P$ the orthogonal projection matrix, and here too it holds that $\hat Y = PY$

A square matrix is a projection matrix if and only it is equal to its square.

You know that $P$ is a projection matrix, therefore (multiply it out), $P^2 = P$. Therefore, $P_2^2 = P_2$. Therefore $P_2$ is a projection matrix.(as is $P$).

Exercise for you: What is the relation between the subspaces to which $P$ and $P_2$ project?

• They are orthogonal to each other. I think you incorrectly understood my question(sorry for that). I know, that $P_2$ is projection matrix. But the question is: why do we use $P_2$ in regression analysis(in my linear model class we are using only $P_2$, instead of $P$? Which purposes have $P$ and $P_2$? – Daniel Yefimov Nov 25 '16 at 18:52
• Read pp. 5-7 as precursor, then pp. 8-10 of stat.cmu.edu/~ryantibs/datamining/lectures/13-reg1.pdf . – Mark L. Stone Nov 25 '16 at 19:02
• is that another way of stating idempotency or is there an important reason why you define a projection matrix as equal to its own square, rather than saying: A projection matrix is idempotent? – Beyer Nov 25 '16 at 20:39
• Idempotence of a matrix is a fancy way of saying that it equals its own square. I decided not to use terminology (a "big" word) not everyone would know - that was a judgment call - I'm just a working class Joe at heart. – Mark L. Stone Nov 25 '16 at 21:06