18
$\begingroup$

I am quite new to this, so I hope you forgive me if the question is naïve. (Context: I am learning econometrics from Davidson & MacKinnon's book "Econometric Theory and Methods", and they do not seem to explain this; I've also looked at Luenberger's optimization book that deals with projections at an a bit more advanced level, but with no luck).

Suppose that I have an orthogonal projection $\mathbb P$ with is associated projection matrix $\bf P$. I am interested in projecting each vector in $\mathbb{R}^n$ into some subspace $A \subset \mathbb{R}^n$.

Question: why does it follow that $\bf{P}=P$$^T$, that is, $\bf P$ is symmetric? What textbook could I look at for this result?

$\endgroup$
1
16
$\begingroup$

This is a fundamental results from linear algebra on orthogonal projections. A relatively simple approach is as follows. If $u_1, \ldots, u_m$ are orthonormal vectors spanning an $m$-dimensional subspace $A$, and $\mathbf{U}$ is the $n \times p$ matrix with the $u_i$'s as the columns, then $$\mathbf{P} = \mathbf{U}\mathbf{U}^T.$$ This follows directly from the fact that the orthogonal projection of $x$ onto $A$ can be computed in terms of the orthonormal basis of $A$ as $$\sum_{i=1}^m u_i u_i^T x.$$ It follows directly from the formula above that $\mathbf{P}^2 = \mathbf{P}$ and that $\mathbf{P}^T = \mathbf{P}.$

It is also possible to give a different argument. If $\mathbf{P}$ is a projection matrix for an orthogonal projection, then, by definition, for all $x,y \in \mathbb{R}^n$ $$\mathbf{P}x \perp y-\mathbf{P}y.$$ Consequently,
$$0 = (\mathbf{P} x)^T (y - \mathbf{P}y) = x^T \mathbf{P}^T (I - \mathbf{P}) y = x^T (\mathbf{P}^T - \mathbf{P}^T \mathbf{P}) y $$ for all $x, y \in \mathbb{R}^n$. This shows that $\mathbf{P}^T = \mathbf{P}^T \mathbf{P}$, whence $$\mathbf{P} = (\mathbf{P}^T)^T = (\mathbf{P}^T \mathbf{P})^T = \mathbf{P}^T \mathbf{P} = \mathbf{P}^T.$$

$\endgroup$
5
  • $\begingroup$ Thanks for your insightful comment(s)! Somehow the Wikipedia article, which mentioned something about self-adjointness of the projection operator threw me off, as your proofs are not that difficult. :) BTW, do you have a favorite linear algebra text that deals with this kind of stuff? $\endgroup$ – weez13 Nov 7 '11 at 22:06
  • $\begingroup$ The elementary linear algebra book I know the best does not cover this. The best references I know are advanced books on functional analysis. The Linear algebra done right book looks good, but I don't know it. $\endgroup$ – NRH Nov 8 '11 at 10:01
  • $\begingroup$ A note: NRH's answer assumes that $x = x^T$. That is, the only case in which $(Px)^T = xP^T$ (as claimed in the equality $(Px)^T(y-Py)=xP^T(I-P)y$) is when $x = x^T$ because for any linear map $P$ and vector $x$, $$(Px)^T = x^TP^T.$$ This does not affect really the result of the proof, as the implication that $P^T - P^TP = 0$ holds in both cases, but I thought it would be worth mentioning. $\endgroup$ – Milan Mosse Nov 3 '16 at 20:47
  • 1
    $\begingroup$ @Milan Thank you for noticing. Observe that $x=x^T$ for $x\in\mathbb{R}^n$ is possible only when $n=1$, which is uninteresting. What happened simply is that a few transposes were lost on the $x$'s in the penultimate line. I have restored the missing transposes to make the algebra correct. $\endgroup$ – whuber Nov 3 '16 at 21:30
  • $\begingroup$ It is not enough characters for me to be able to edit it, but $\mathbf{U}$ is an $n \times m$ matrix; not an $n \times p$ matrix. Not sure where $p$ came from. $\endgroup$ – kanso37 Jul 28 '20 at 2:41
3
$\begingroup$

An attempt at geometrical intuition... Recall that:

  1. A symmetric matrix is self adjoint.
  2. A scalar product is determined only by the components in the mutual linear space (and independent of the orthogonal components of any of the vectors).

What you want to "see" is that a projection is self adjoint thus symmetric-- following (1). Why is this so? Consider the scalar product of a vector $x$ with the projection $A$ of a second vector $y$: $ \langle x,Ay \rangle$. Following (2), the product will depend only on the components of $x$ in the span of the projection of $y$. So the product should be the same as $\langle Ax,Ay \rangle$, and also $\langle Ax,y\rangle $ following the same argument.

Since $A$ is self adjoint- it is symmetric.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks a lot! Before reading your comment, I was quite confused about why self-adjointness is crucial here. Now I have some clue, thanks! $\endgroup$ – weez13 Nov 7 '11 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.