1
$\begingroup$

I am trying to get some intuition regarding the U and V matrices in SVD ($M=UEV^T$). I think these are orthonormal basis vectors, but I am struggling to get an intuition if they represent anything more than matrix transformations.

For example, in NMF, I understand that the decomposed matrices can be combined as a linear combination of basis vectors in one matrix with weights in the other matrix. In PCA, I understand that the eigenvectors have real significance in representing orthogonal axes of maximal variation, defined by the eigenvalues. But in SVD, I don't see any immediate connection.

Could someone enlighten me?

$\endgroup$
0
0
$\begingroup$

In a singular value decomposition, $M$ is not (necessarily) symmetric or even square; it's a transformation from one space ($\mathbb{R}^M$) to another ($\mathbb{R}^n$).

Let's supppose $m>n$. $M$ can be decomposed into a transformation into a convenient basis for $\mathbb{R}^n$ by $U$, then a projection and scaling by $E$ into a basis for $\mathbb{R}^n$ , then a rotation into the target basis in $\mathbb{R}^n$ by $V^T$.

(Alternatively, for the compact SVD, $U$ is $m\times n$ and includes the projection, and $E$ is just the scaling)

Also, just as in PCA, it's no loss of generality (and is standard) to organise $E$ from largest to smallest singular value, and $U$ and $V$ represent orthogonal axes of maximal variation in the two spaces. There are various efficient algorithms for giving you just the largest few singular values and corresponding vectors (either Lanczos-type or stochastic algorithms)

$\endgroup$
1
  • $\begingroup$ "Also, just as in PCA, it's no loss of generality (and is standard) to organise 𝐸 from largest to smallest singular value, and 𝑈 and 𝑉 represent orthogonal axes of maximal variation in the two spaces" Would you mind elaborating? This isn't immediately obvious to me. In PCA, I saw how this would be the case because of the objective function (stats.stackexchange.com/questions/217995/…), but here I'm not so sure $\endgroup$ – Victor M Mar 22 at 5:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.