Intuition using linear algebra that the rank of the projection matrix equals the rank of the design matrix Using linear algebra to explain, can someone show the intuition? I can show that the ranks are the same by using properties of rank but can't get my head around the whole projection thing more than just thinking hazily.
 A: Let the number of observations be $n$, let $p$ count the parameters, and let $r$ designate the rank of the $n\times p$ design matrix $X$ (which, by definition, is the dimension of the image of $X$).
The SVD
A Singular Value Decomposition expresses $X$ as a product
$$X = U\Sigma V^\prime$$
where the matrices $U$ (dimensions $n\times r$) and $V$ (dimensions $p \times r$) are orthogonal and $\Sigma$ is an $r\times r$ diagonal matrix with no zeros.  A nonzero $X$ always has an SVD.
(Here's one proof: the columns of $V$ must be the eigenvectors of $X^\prime X$ corresponding to nonzero eigenvalues while the columns of $U$ must be the eigenvectors of $XX^\prime$ corresponding to nonzero eigenvalues.  Those eigenvectors and eigenvalues exist because both $X^\prime X$ and $XX^\prime$ are nonzero real symmetric matrices: this is part of the Spectral Theorem.  Although in the SVD it is arranged that all elements of $\Sigma$ be nonnegative, we won't need that here.)
Interpreting the SVD
One way to view the SVD is that it expresses the columns of $X$ as linear combinations of the columns of $U$: the coefficients are the columns of $\Sigma V^\prime$.  You may therefore think of $U$ as being an orthonormal frame for the image of $X$, which is an $r$-dimensional subspace $\mathbb W\subset \mathbb{R}^n$. ("Orthonormal" means "orthogonal" and of unit length; "orthogonal" means mutually perpendicular, which is a crucial simplification.) Indeed, it is appealing to consider this geometrically: upon choosing bases for all the vector spaces in question, for $\beta\in\mathbb{R}^p$, $X$ determines a linear transformation from $\mathbb{R}^p$ into $\mathbb{R}^n$ in three steps:

*

*$\beta \to V^\prime \beta$ is a vector in $\mathbb{R}^r$.


*$\Sigma$ rescales each of the $r$ basis vectors of $\mathbb{R}^r$.


*The resulting $r$ coefficients determine a linear combination of the columns of $U$: that is, a unique vector in $\mathbb W$.  (Equivalently, the original $r$ coefficients $V^\prime \beta$ specify linear combinations of the orthogonal columns of $U\Sigma$.)
The image of $\beta$ in step (1) consists of all vectors spanned by the $r$ rows of $V$, and therefore has dimension $r$.  Because the diagonal elements of $\Sigma$ are nonzero, the rescaling in (2) does not change that dimension.  Thus the dimension of the space generated in (3) is also $r$.  Consequently, the rank of $X$ is $r$.
In statistical language, $V$ finds identifiable linear combinations of the parameters $\beta$ and the diagonal elements of $\Sigma$ establish scale factors in the space $\mathbb W$ spanned by the columns of $X$, which is the space of all possible vectors $y$ that can be exactly represented as linear combinations of those columns.
More About Projections
Here's a related algebraic argument.
Any orthonormal frame $U$ determines a projection matrix $UU^\prime$.  Specifically, left multiplying any vector $y\in\mathbb{R}^n$ by $U^\prime$ computes the coefficients of $y$ for each of the columns of $U$.  Obviously this has rank $r$: since the columns of $U$ each get projected to themselves, the image of the linear transformation $UU^\prime$ is precisely $\mathbb W$.
You probably know of a different looking formula for the "projection matrix": namely, $P=X(X^\prime X)^{-} X^\prime$ where $(X^\prime X)^{-}$ is a generalized inverse of $X^\prime X$.  Using the SVD we may simplify this:
$$P = (U\Sigma V^\prime)((U\Sigma V^\prime)^\prime\, (U\Sigma V^\prime))^{-} (U\Sigma V^\prime)^\prime = UU^\prime.$$
This is because terms of the form $V^\prime V=I_r=U^\prime U$ are identity matrices, which disappear in the multiplications, and the generalized inverse of $\Sigma^2$ is just $\Sigma ^{-2}$.
It is now obvious that $P$ has rank $r$.
