4
$\begingroup$

I was working on a question on projection matrix. Since, projection matrix is idempotent, symmetric and square matrix, it must always be equal to $I$ (Identity matrix). This can be shown by multiplying the inverse of projection matrix on both the sides. If it is equal to $I$, then I do not understand the point of using it. Can anyone please explain?

$\endgroup$
  • 2
    $\begingroup$ Can you show the linear algebra behind your question? The "multiplying the inverse of projection matrix on both sides." $\endgroup$ – Antoni Parellada Aug 22 '16 at 18:24
  • 3
    $\begingroup$ Steve, check this out. You are talking about the only scenario where the projection matrix is invertible. Hope it helps. $\endgroup$ – Antoni Parellada Aug 22 '16 at 20:02
  • 2
    $\begingroup$ 2D Matrix with either both rows [1,0] or [0,1] or both columns $[1,0]^T$ or $[0,1]^T$ are all idempotent projection matrices, and all are singular. Also the matrix of all zeros. $\endgroup$ – Mark L. Stone Aug 22 '16 at 20:38
  • 2
    $\begingroup$ I have tried to make the title clearer (remember that many people will see the title in a link, but not the content of the question itself, so "econometrics question" isn't very useful as a title). If you don't like my suggested title, feel free to edit it to something better. $\endgroup$ – Silverfish Aug 22 '16 at 22:32
  • 1
    $\begingroup$ Any zero square matrix satisfies the same constraints and isn't an identity matrix! $\endgroup$ – Mehrdad Aug 23 '16 at 0:03
10
$\begingroup$

Just to illustrate. If you are making reference to the unregularized OLS projection matrix, here is a "real life" example showing that there has to be a misconception in your OP. We are regressing miles-per-gallon over vehicle weight of the mtcars dataset, and choosing (to be able to copy and paste), the first 5 rows of data only:

dat = mtcars[5,]    
dat
                   mpg cyl disp  hp drat    wt  qsec vs am gear carb
Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2

Now let's generate manually the projection matrix. First the model matrix and y:

fit = lm(mpg ~ wt, data = dat)
X = model.matrix(fit)
y = dat[,1]

Now the projection matrix:

Pr = X %*% ((solve(t(X) %*% X)) %*% t(X))

Idempotent?

all.equal(Pr,Pr%*%Pr)
[1] TRUE

Check!

Symmetric?

all.equal(Pr, t(Pr))
[1] TRUE

Check!

Square?

 dim(Pr)
[1] 5 5

Check!

Equal to the identity matrix? Well, no... Here is Pr:

                   Mazda RX4 Mazda RX4 Wag  Datsun 710 Hornet 4 Drive Hornet Sportabout
Mazda RX4         0.29313831     0.2064585  0.39511457     0.09088541        0.01440322
Mazda RX4 Wag     0.20645850     0.2004479  0.21352984     0.19243366        0.18713015
Datsun 710        0.39511457     0.2135298  0.60874366    -0.02858313       -0.18880494
Hornet 4 Drive    0.09088541     0.1924337 -0.02858313     0.32783133        0.41743273
Hornet Sportabout 0.01440322     0.1871301 -0.18880494     0.41743273        0.56983885

Following up on the reply with additional clarification in the OP (as well as now the change in the title), the argument was $ P'P=P$; $P'PP^{-1}=PP^{-1}$; hence $ P'=I$. But the orthogonal projection matrix is not invertible, as nicely explained here. So the equations would only hold in the case of the identity matrix. In fact, it's easy to picture it: $P$ will project vectors orthogonal to the column space of $ X$ (think error terms) into zero - hence, a projection matrix will have a null space; and hence they are not invertible.

For an identity matrix with column vectors in $\mathbb R^n$, each vector will be projected not onto a subspace of $\mathbb R^n$, but onto $\mathbb R^n$ itself. It can be considered a special form of projection matrix, and the only one invertible.

$\endgroup$
8
$\begingroup$

There are a couple excellent answers already, but I'd like to add one more perspective (hinted at in @Antoni's answer).

An idempotent matrix satisfies the matrix equation

$$ X^2 = X $$

or

$$ X^2 - X = 0 $$

Which we can factor

$$ X(X - I) = 0 $$

This means there are relatively few possibilities for the minimal polynomial of $X$

The Minimal Polynomial is $p(x) = x$:

In this case $X = 0$ immediately, we have the zero matrix. Not very interesting.

The Minimal Polynomial is $p(x) = x - 1$:

In this case $X - I = 0$, so $X = I$. This is the case you are thinking of.

The Minimal Polynomial is $p(x) = x(x - 1)$:

This is the interesting "in between" case.

Because the minimal polynomial splits into linear factors, the matrix $X$ is diagonalizable, with only $0$ and $1$'s on the resulting diagonal.

Another way to look at this: $X$ has a full linearly independent set of eigenvectors, say

$$ e_1, e_2, \ldots, e_{k_1}, f_1, f_2, \ldots, f_{k_2} $$

With $k_1 + k_2 = n$. Some of the eigenvectors have an associated eigenvalue of zero

$$ X e_1 = 0, X e_2 = 0, \ldots, X e_{k_1} = 0$$

and the rest have an eigenvalue of one

$$ X f_1 = f_1, X f_2 = f_2, \ldots, X f_{k_2} = f_{k_2}$$

Now a geometric picture emerges, and it justifies the name projection matrix. If we look at the subspace spanned by the $f$'s, then the image of any vector is in this subspace

$$ X (a_1 e_1 + \cdots + a_{k_1} e_{k_1} + b_1 f_1 + \cdots + b_{k_2} f_{k_2}) = b_1 f_1 + \cdots + b_{k_2} f_{k_2} $$

and, by the same calculation, the mapping restricted to this subspace is indeed the identity. You see, the map is really a projection into the subspace spanned by the $f$'s.

$\endgroup$
5
$\begingroup$

An idempotent matrix https://en.wikipedia.org/wiki/Idempotent_matrix (see also https://en.wikipedia.org/wiki/Projection_(linear_algebra) ) is a matrix which equals its square. So $P$ being idempotent means that $P^2 = P$. The identity matrix is idempotent, but is not the only such matrix.

Projection matrices need not be symmetric, as the the 2 by 2 matrix whose rows are both $[0,1]$, which is idempotent, demonstrates. This provides a counterexample to your claim. By (pre-)multiplying both dies of $P^2 = P$, you have lost solutions.

Edit: In response to the comment by @Antoni Parellada , the OLS projection matrix $X(X^TX)^{-1}X^T$ is idempotent and symmetric, but in general it is not equal to the identity matrix, although it is possible that it could equal the identity matrix in some special (unusual) case.

$\endgroup$
  • 2
    $\begingroup$ If he's making reference to the OLS projection matrix $X(X^TX)^{-1}X^T$, it is symmetric. Can you answer the question more generally? $\endgroup$ – Antoni Parellada Aug 22 '16 at 18:11
  • 1
    $\begingroup$ Mark, Of course it is not equal to the identity. That is not what I meant. It is symmetrical, but naturally, it does contain information. I just think that the premise of the OP is incorrect. $\endgroup$ – Antoni Parellada Aug 22 '16 at 18:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.