What do the matrix (S, U, V) returned by singular value decomposition represent (in terms of variation)?

Question

I believe SVD on a matrix A returns three matrices: U, S, and V.

Let's imagine A is a data matrix with training examples/records/whatever you call them as its rows and attributes as its columns.

I think S is a diagonal matrix, where the $i$-th diagonal value is the variation in the $i$-th attribute (column) of the matrix A. Furthermore, the diagonal values of S decrease as you go left to right/top to bottom (the matrix is sorted).

I think U says something about the records themselves. I believe each row represents one record. I often see the first two columns U graphed such that the x axis is U1 (the first column) and the y axis is U2, but I don't know what the resulting graph is telling us.

I haven't been able to figure out what V does.

Is my understanding of S correct? And what do U and V represent? Any help is appreciated!

Answer 1

By definition: Given that an $m\times n$ matrix $A$ has rank $r$, $A$ can be factored $A=U\times S\times V^T$, where $U$ and $V$ are orthogonal matrices containing the singular vectors. We can think of $U$ and $V$ as rotations and reflections and $S$ as the stretching matrix. Since $V$ is an orthogonal matrix $(𝐕^⊤𝐕=𝐈)$, $AA^T=(USV^T)(VSU^T)=USU^T$, where $S$ has all the eigenvalues, and $U$ has its eigenvectors.

If you really wonder what U, S, and V do for the A see the following figure from http://web.cs.iastate.edu/~cs577/handouts/svd.pdf