Definition of orthogonal matrix I am reading the book Elements of Statistical Learning and trying to understand singular value decomposition (SVD). In particular, what is an orthogonal matrix as it relates to SVD?
According to Wikipedia, an orthogonal matrix is a square matrix, the transpose of an which is equal to its inverse.
https://en.wikipedia.org/wiki/Orthogonal_matrix
As this CrossValidated question points out however, in the SVD example discussed in Elements of Statistical Learning, this does not hold.
SVD in linear regression
What am I missing?
 A: The formulations are equivalent.
By transposing $X$ if necessary, we may reduce the situation to where $X$ has at least as many rows, $n$, as columns, $p$.  Consider the decomposition of $X$ into
$$X = U\Sigma V^\prime$$
for an $n\times n$ orthogonal matrix $U$, an $n\times p$ matrix $\Sigma$ that is diagonal in the sense that $\Sigma_{ij}=0$ whenever $i\ne j$, and a $p\times p$ orthogonal matrix $V$.  This $\Sigma$ can be considered to be a diagonal $p\times p$ matrix $S$ stacked on top of a $(n-p)\times p$ matrix of zeros, $Z$.  The effect of $Z$ in the product $U\Sigma$ is to "kill" the last $n-p$ columns of $U$.  We may therefore drop those columns and drop $Z$, producing a decomposition
$$X = U_0 S V^\prime$$
where the columns of $U_0$--being the first $p$ columns of $U$--are orthogonal.  The dimensions of these matrices are $n\times p$, $p\times p$, and $p\times p$.
Conversely--there's a theorem involved here--we may always extend an $n\times p$ matrix $U_0$ of orthogonal (and unit length) columns into an orthogonal $n\times n$ matrix.  Geometrically this is obvious--you can always complete a partial basis of $p$ unit length, mutually perpendicular vectors into a full basis of such vectors--and algebraically it is performed using the Gram-Schmidt process.  ($U$ is not unique unless $n=p$.)  After doing this, stack an $n-p\times p$ matrix of zeros underneath $S$ to produce $\Sigma$ and you have recovered the first decomposition.

A more modern conception of matrices and their multiplication is that any $a\times b$ matrix $Y$ represents a linear transformation $T_Y:V^{(b)}\to V^{(a)}$ where $V^{(a)}$ is a vector space of dimension $a$ and $V^{(b)}$ a vector space of dimension $b$, and multiplication represents functional composition.  The matrix $V^\prime$ maps $\mathbb{R}^p$ to itself, $\Sigma$ maps $\mathbb{R}^p$ into the subspace of $\mathbb{R}^n$ spanned by the first $p$ components, and $U$ maps $\mathbb{R}^n$ into itself.  Since the image of $V^\prime$ is only this $p$-dimensional subspace, $U$ and its restriction to the subspace (called $U_0$ above) agree on the image of $\Sigma$.  Moreover, $\Sigma$ decomposes as a linear transformation $S:\mathbb{R}^p\to\mathbb{R}^p$ followed by a fixed embedding of $\mathbb{R}^p$ into $\mathbb{R}^n$. (The matrix of this embedding is a diagonal $n\times p$ matrix with all ones on the diagonal.)  This makes it geometrically obvious that the transformations from $\mathbb{R}^p$ to $\mathbb{R}^n$ represented by $U_0 S V^\prime$ and $U\Sigma V^\prime$ are the same.
