From Elements of Statistical Learning (pp. 64, 66), they explain how the $N \times p$ data matrix $X$ can be written as $$X = UDV^{T}$$

Here $U$ and $V$ are $N \times p$ and $p \times p$ orthogonal matrices, with the columns of $U$ spanning the column space of $X$, and the columns of $V$ spanning the row space. $D$ is a $p \times p$ diagonal matrix, which diagonal entries $d_1 \geq d_2 \geq \cdots \geq d_p \geq 0$ called the singular values of $X$.

Then they go on to say that

$$X\boldsymbol{\hat{\beta}}^{\text{ls}} = X(X^{T}X)^{-1}X^{T}Y = UU^{T}Y$$

($\boldsymbol{\hat{\beta}}^{\text{ls}}$ being the least squares estimate).

What doesn't make sense to me:

  1. $U$ is $N \times p$ and is orthogonal. So $U^{T} = U^{-1}$. But last I checked, only square matrices are invertible.
  2. My attempt at the derivation: $$\begin{align} X\boldsymbol{\hat{\beta}}^{\text{ls}} &= X(X^{T}X)^{-1}X^TY \\ &= UDV^{T}(VD^{T}U^{T}UDV^{T})^{-1}VD^{T}U^{T}Y \\ &= UDV^{T}[(VD^{T})(U^{T}U)(DV^{T})]^{-1}VD^{T}U^{T}Y \\ &= UDV^{T}(DV^{T})^{-1}(U^{T}U)^{-1}(VD^{T})^{-1}VD^{T}U^{T}Y \\ &= U(U^{T}U)^{-1}U^{T}Y \end{align}$$ What did I do wrong here? [Maybe it's unreasonable to assume $U^{T}U$ is invertible, but I wouldn't know what else to do here.]
  • 4
    $\begingroup$ By definition of the SVD, $U^\prime U$ is the identity matrix. $\endgroup$ – whuber Dec 10 '15 at 20:23
  • $\begingroup$ @whuber Ah, so that's what "unitary" means (on Wikipedia). Thank you. Am I allowed to assume $DV^{T}$ (and $VD^{T}$ as well) are invertible? (Sorry, I am new to using the SVD.) $\endgroup$ – Clarinetist Dec 10 '15 at 20:27
  • $\begingroup$ @whuber I suppose they might be, after giving it some thought. $D$ is a diagonal matrix, so it's invertible, and $V$ is square and orthogonal (thus invertible), so I would imagine that a product (and any transpositions involved) would be invertible as well. $\endgroup$ – Clarinetist Dec 10 '15 at 20:28
  • $\begingroup$ Re: invertibility of the product, $\det(AB) = \det(A)\det(B)$ so if both are invertible then the product is too $\endgroup$ – jld Dec 10 '15 at 20:31
  • $\begingroup$ $D$ is invertible if and only if $d_p$ is nonzero. This is an implicit assumption in the second quotation. $\endgroup$ – whuber Dec 10 '15 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.