Elements of Statistical Learning, p 66

The SVD of the $N \times p$ matrix $X$ has the form $X = UDV^T$

Here $U$ and $V$ are $N \times p$ and $p \times p$ orthogonal matrices, with the columns of $U$ spanning the column space of $X$, and the columns of $V$ spanning the row space.

$D$ is a $p \times p$ diagonal matrix, with diagonal entries $d_1 \ge d_2 \ge ... \ge d_p \ge 0$ called the singular values of $X$. If one or more values $d_j = 0$, $X$ is singular.

Using the singular value decomposition we can write the least squares fitted vector as :

$X \hat{\beta_{ls}} = X(𝑋^T𝑋)^{−1}𝑋^Ty = UU^Ty$

The demonstration is available here p 266, and makes total sense.

Yet, as $U$ is orthogonal, it seems to me that it implies that $UU^Ty=Iy=y$, and thus that $X\hat{\beta_{ls}}=y$, which makes not sense. Indeed, it would mean that the the residual vector $y-X\hat{\beta_{ls}}$ is null. But this should only true in the specific case where $y$ can be expressed as a linear combination of the columns of $X$.

It seems to me that i'm totally missing something but i don't know what it is !

  • $\begingroup$ I just updated with an example to show why it's not possible $\endgroup$ – jld Mar 19 '20 at 23:33
  • 1
    $\begingroup$ Please visit stats.stackexchange.com/help/merging-accounts to merge your accounts, Tim, so you can edit your question freely. $\endgroup$ – whuber Mar 20 '20 at 1:53

$U$ has orthogonal columns but it's not square and isn't an orthogonal matrix. $UU^T \neq I$ in general since it only has rank $p$ and $UU^T$ involves dot products of the rows which don't have to be orthogonal when $p<n$. If $U$ was square then it would be the case that $UU^T = I$ which makes sense since then $\hat y = y$ since $X$ is a bijection.

I'm updating to add this example of why this isn't possible if $n > p$.

Let $$ U = \left(\begin{array}{cc} a & b\\ c&d \\ e&f\end{array}\right). $$

Suppose $U^TU = I_2$ and $UU^T = I_3$. These turn into the following 9 constraints: $$ a^2+c^2+e^2 = b^2+d^2+f^2 = 1 \hspace{2cm}(1)\\ ab + cd + ef = 0 \\ a^2 + b^2 = c^2 + d^2 = e^2 + f^2 = 1 \hspace{2cm}(2)\\ ac + bd = ae + bf = ce + df = 0 $$ But we only have 6 variables, and as it turns out these are not satisfiable. To see this, note that we have $$ a^2 + b^2 + c^2 + d^2 + e^2 +f^2 = 2 $$ from adding the constraints in (1) together, but adding the constraints in line (2) together gives $$ a^2 + b^2 + c^2 + d^2 + e^2 +f^2 = 3 $$ which is impossible. Thus unless $n=p$ we don't have enough freedom to make $U^TU = I$ while also having $UU^T = I$. Since $U^TU = I$ is guaranteed by construction, the result is that $UU^T \neq I$ unless $n=p$.

I think interpreting this can also help with this. I like to interpret $\hat y = UU^Ty$ as follows: we can expand $U$ to an $n\times n$ matrix $\tilde U = (U \mid U_\perp)$ that actually does contain an orthonormal basis for $\mathbb R^n$ (this can be done via Gram-Schmidt). In this basis we'll have $y = \tilde U \tilde z$ where $\tilde z \in \mathbb R^n$ is $y$'s coordinate w.r.t. $\tilde U$. I'll partition $\tilde z = {z \choose z_\perp}$ where $z \in \mathbb R^p$ gives the coordinates for the basis vectors in $U$, and $z_\perp$ analogously gives the coordinates for the basis vectors in $U_\perp$.

This means $$ y = \tilde U \tilde z = Uz + U_\perp z_\perp $$ so $$ U^Ty = U^TUz + U^TU_\perp z_\perp = z $$ therefore $U^Ty$ picks out the coordinates of $y$ w.r.t. just the basis vectors in $U$. And then $UU^Ty = Uz$ gives the corresponding column space element in $\mathbb R^n$ with just those coordinates. So it's like doing a change of basis but only keeping the coordinates from $U$, and then getting the column space vector specified by those coordinates.

This reflects the fact that when we're talking about a $p$-dimensional subspace in $\mathbb R^n$, we can think of the actual vectors in $\mathbb R^n$, which are like $X\beta$, or we can think about the coordinates indexing the column space which is like $\beta$.


It says that $U \in \mathbb{R}^{n \times p}$ so the columns are orthogonal but the matrix is not necessarily orthogonal in the sense that $UU^T =I$ ($\dim(UU^T) =p$)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.