Please confirm or reject my line of reasoning:

Given SVD of $X$: $X_{NxP}=U_{NxP}D_{PxP}V_{PxP}'$,

Variance along ith column vector of $U$ is given by $||X'u_i||^2=u'_iXX'u_i=u'_id_i^2u_i=d_i^2$, where $d_i$ is ith singular value and second equality comes from eigendecomposition of $XX'=UD^2U'$. I omit division by $N$ as it doesn't affect the result.

Variance along ith column vector of $V$ is given by $||Xv_i||^2=v'_iX'Xv_i=v'_id_i^2u_i=d_i^2$

But in such a case the variance along principal components (columns of $V$) and variance along left singular vectors (columns of $U$) is equal up to a factor (N or P). Is this true?

If this is not the case, how can I argue that lower singular value implies lower variance along corresponding left singular vector (relative to other left singular vectors)? In particular, I want to confirm the following passage from Elements of Statistical Learning, p.67:

small singular values dj correspond to directions in the column space of X having small variance, and ridge regression shrinks these directions the most.

Where ridge regression with respect to SVD is given by: $$X\beta^{Ridge}=X(X'X +\lambda I)^{-1}X'y=UD(D^2+'\lambda I)^{-1}DU'y=\sum_{j=1}^{P}u_j\frac{d_j^2}{d_j^2+\lambda}u_j'y$$

It is clear that this is the case for principal components $v_j$ (proven in text) but the above expression shrinks left singular vectors $u_j$ thus the question.

  • 1
    $\begingroup$ Indeed, that is true. The "variance" (in your sense, not in the sense of Var(X) where X is an r.v.) is encoded in the singular values. The U and V matrices have orthonormal columuns, in particular, they all have unit $\|\cdot\|_2$-norm. $\endgroup$ – Michael Oct 6 '20 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.