Using linear algebra to explain, can someone show the intuition? I can show that the ranks are the same by using properties of rank but can't get my head around the whole projection thing more than just thinking hazily.


Let the number of observations be $n$, let $p$ count the parameters, and let $r$ designate the rank of the $n\times p$ design matrix $X$ (which, by definition, is the dimension of the image of $X$).


A Singular Value Decomposition expresses $X$ as a product

$$X = U\Sigma V^\prime$$

where the matrices $U$ (dimensions $n\times r$) and $V$ (dimensions $p \times r$) are orthogonal and $\Sigma$ is an $r\times r$ diagonal matrix with no zeros. A nonzero $X$ always has an SVD.

(Here's one proof: the columns of $V$ must be the eigenvectors of $X^\prime X$ corresponding to nonzero eigenvalues while the columns of $U$ must be the eigenvectors of $XX^\prime$ corresponding to nonzero eigenvalues. Those eigenvectors and eigenvalues exist because both $X^\prime X$ and $XX^\prime$ are nonzero real symmetric matrices: this is part of the Spectral Theorem. Although in the SVD it is arranged that all elements of $\Sigma$ be nonnegative, we won't need that here.)

Interpreting the SVD

One way to view the SVD is that it expresses the columns of $X$ as linear combinations of the columns of $U$: the coefficients are the columns of $\Sigma V^\prime$. You may therefore think of $U$ as being an orthonormal frame for the image of $X$, which is an $r$-dimensional subspace $\mathbb W\subset \mathbb{R}^n$. ("Orthonormal" means "orthogonal" and of unit length; "orthogonal" means mutually perpendicular, which is a crucial simplification.) Indeed, it is appealing to consider this geometrically: upon choosing bases for all the vector spaces in question, for $\beta\in\mathbb{R}^p$, $X$ determines a linear transformation from $\mathbb{R}^p$ into $\mathbb{R}^n$ in three steps:

  1. $\beta \to V^\prime \beta$ is a vector in $\mathbb{R}^r$.

  2. $\Sigma$ rescales each of the $r$ basis vectors of $\mathbb{R}^r$.

  3. The resulting $r$ coefficients determine a linear combination of the columns of $U$: that is, a unique vector in $\mathbb W$. (Equivalently, the original $r$ coefficients $V^\prime \beta$ specify linear combinations of the orthogonal columns of $U\Sigma$.)

The image of $\beta$ in step (1) consists of all vectors spanned by the $r$ rows of $V$, and therefore has dimension $r$. Because the diagonal elements of $\Sigma$ are nonzero, the rescaling in (2) does not change that dimension. Thus the dimension of the space generated in (3) is also $r$. Consequently, the rank of $X$ is $r$.

In statistical language, $V$ finds identifiable linear combinations of the parameters $\beta$ and the diagonal elements of $\Sigma$ establish scale factors in the space $\mathbb W$ spanned by the columns of $X$, which is the space of all possible vectors $y$ that can be exactly represented as linear combinations of those columns.

More About Projections

Here's a related algebraic argument.

Any orthonormal frame $U$ determines a projection matrix $UU^\prime$. Specifically, left multiplying any vector $y\in\mathbb{R}^n$ by $U^\prime$ computes the coefficients of $y$ for each of the columns of $U$. Obviously this has rank $r$: since the columns of $U$ each get projected to themselves, the image of the linear transformation $UU^\prime$ is precisely $\mathbb W$.

You probably know of a different looking formula for the "projection matrix": namely, $P=X(X^\prime X)^{-} X^\prime$ where $(X^\prime X)^{-}$ is a generalized inverse of $X^\prime X$. Using the SVD we may simplify this:

$$P = (U\Sigma V^\prime)((U\Sigma V^\prime)^\prime\, (U\Sigma V^\prime))^{-} (U\Sigma V^\prime)^\prime = UU^\prime.$$

This is because terms of the form $V^\prime V=I_r=U^\prime U$ are identity matrices, which disappear in the multiplications, and the generalized inverse of $\Sigma^2$ is just $\Sigma ^{-2}$.

It is now obvious that $P$ has rank $r$.

  • $\begingroup$ This is great stuff! Can I think about it as each entry in the dependent variable needs to be modified by the projection matrix by each on of the vectors on a basis of the column space of the model matrix for the final projection to inhabit the vector space of the model matrix - hence the cardinality of the column space of any basis of the MM and Prjt. matrices have to agree? $\endgroup$ Feb 7 '17 at 20:26
  • $\begingroup$ @Antoni I believe it's simpler than that. When you have an orthonormal frame for a vector subspace, the projection of any vector $y$ (the "dependent variable") into that subspace can be found by separately finding the component of $y$ on each element of the frame. This is the underlying idea behind orthogonal polynomials and various experimental design matrices, too: when all independent variables are uncorrelated (and thereby can serve as the columns of $U\Sigma$), the coefficient of each can be found without reference to any of the others. $\endgroup$
    – whuber
    Feb 7 '17 at 20:44
  • $\begingroup$ That is not a proof of the existence of the SVD. $\endgroup$ Feb 7 '17 at 22:17
  • $\begingroup$ @Federico Although it's no proof, it's a sketch of one. Where do you think it falls short or is invalid? $\endgroup$
    – whuber
    Feb 7 '17 at 22:35
  • $\begingroup$ @whuber There may be multiple choices for $U$ and $V$. For instance, if $X$ is orthogonal, $X'X=XX'=I$, so any choice of orthogonal $U$ and $V$ works. Not all these choices, though, produce a diagonal $U'XV$, which you need to complete the proof. $\endgroup$ Feb 8 '17 at 7:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.