5
$\begingroup$

I am reading the book Elements of Statistical Learning and trying to understand singular value decomposition (SVD). In particular, what is an orthogonal matrix as it relates to SVD?

According to Wikipedia, an orthogonal matrix is a square matrix, the transpose of an which is equal to its inverse. https://en.wikipedia.org/wiki/Orthogonal_matrix

As this CrossValidated question points out however, in the SVD example discussed in Elements of Statistical Learning, this does not hold. SVD in linear regression

What am I missing?

$\endgroup$
1
  • $\begingroup$ I think you are making a mistake about definition of orthogonal matrices. A matrix $X_{m\times n}$ called orthogonal if satisfies, $XX^T=I_m$ and $X^TX=I_n$. refer to page 65 of Matrices and Tensors in Physics. Although this definition for finite matrices result in a square matrix but not necessary all orthogonal matrices are squared. $\endgroup$
    – TPArrow
    Commented Jul 28, 2015 at 8:25

1 Answer 1

5
$\begingroup$

The formulations are equivalent.

By transposing $X$ if necessary, we may reduce the situation to where $X$ has at least as many rows, $n$, as columns, $p$. Consider the decomposition of $X$ into

$$X = U\Sigma V^\prime$$

for an $n\times n$ orthogonal matrix $U$, an $n\times p$ matrix $\Sigma$ that is diagonal in the sense that $\Sigma_{ij}=0$ whenever $i\ne j$, and a $p\times p$ orthogonal matrix $V$. This $\Sigma$ can be considered to be a diagonal $p\times p$ matrix $S$ stacked on top of a $(n-p)\times p$ matrix of zeros, $Z$. The effect of $Z$ in the product $U\Sigma$ is to "kill" the last $n-p$ columns of $U$. We may therefore drop those columns and drop $Z$, producing a decomposition

$$X = U_0 S V^\prime$$

where the columns of $U_0$--being the first $p$ columns of $U$--are orthogonal. The dimensions of these matrices are $n\times p$, $p\times p$, and $p\times p$.

Conversely--there's a theorem involved here--we may always extend an $n\times p$ matrix $U_0$ of orthogonal (and unit length) columns into an orthogonal $n\times n$ matrix. Geometrically this is obvious--you can always complete a partial basis of $p$ unit length, mutually perpendicular vectors into a full basis of such vectors--and algebraically it is performed using the Gram-Schmidt process. ($U$ is not unique unless $n=p$.) After doing this, stack an $n-p\times p$ matrix of zeros underneath $S$ to produce $\Sigma$ and you have recovered the first decomposition.


A more modern conception of matrices and their multiplication is that any $a\times b$ matrix $Y$ represents a linear transformation $T_Y:V^{(b)}\to V^{(a)}$ where $V^{(a)}$ is a vector space of dimension $a$ and $V^{(b)}$ a vector space of dimension $b$, and multiplication represents functional composition. The matrix $V^\prime$ maps $\mathbb{R}^p$ to itself, $\Sigma$ maps $\mathbb{R}^p$ into the subspace of $\mathbb{R}^n$ spanned by the first $p$ components, and $U$ maps $\mathbb{R}^n$ into itself. Since the image of $V^\prime$ is only this $p$-dimensional subspace, $U$ and its restriction to the subspace (called $U_0$ above) agree on the image of $\Sigma$. Moreover, $\Sigma$ decomposes as a linear transformation $S:\mathbb{R}^p\to\mathbb{R}^p$ followed by a fixed embedding of $\mathbb{R}^p$ into $\mathbb{R}^n$. (The matrix of this embedding is a diagonal $n\times p$ matrix with all ones on the diagonal.) This makes it geometrically obvious that the transformations from $\mathbb{R}^p$ to $\mathbb{R}^n$ represented by $U_0 S V^\prime$ and $U\Sigma V^\prime$ are the same.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.