SVD & ICA -- or why doesn't the other rotation matrix in SVD solve for independent components?

Question

When data are a linear mixture of non-gaussian sources, it can be shown that with a rotation, an independent rescaling of each of the rotated axes, and a second rotation you can recover the original, independent axes. Singular value decomposition [SVD] does exactly these three operations -- i.e., it decomposes a matrix X into U, S, and V [i.e, X = USV'], where U & V are rotation matrices. SVD even finds that first rotation [i.e., the PCs] and the scaling for axes [the eigenvalues of the covariance matrix], but the second rotation isn't the rotation to the independent axes. I know the dimensions of the second rotation are wrong, but still...

So here's my [possibly very naive] question: why doesn't SVD find this second rotation to recover the original, independent axes?

There's been a lot posted on how SVD relates to PCA, which makes sense -- but how does SVD relate to ICA? Why can't you recover the unmixing matrix with SVD? or is this just the wrong question to ask...

EDIT:

After reading about this a bit more, I feel like I have a good intuition for why SVD, despite performing the necessary matrix operations [rotate, rescale, rotate], doesn't find the independent components. Both rotation matrices are only sensitive to the correlations in the data -- the U & V matrices are, respectively, the eigenvectors of XX' and X'X. Thus, these two rotations are only sensitive to second-order correlations in the data -- they are not sensitive to higher-order moments [e.g., kurtosis], and these higher-order moments are necessary to find independent projections.

This seems satisfying to me. If others have insights, please let me know!

Answer 1

I think you are (or at least were) confused about several things here.

Doing SVD of your data matrix $\mathbf{X}$ amounts to doing the PCA. Decomposition $\mathbf{X}=\mathbf{USV^\top}$ decomposes the data into principal axes (eigenvectors of the covariance matrix) and principal "scores", or "principal components", i.e. projections of the data onto the principal axes. The "other rotation matrix" you were asking about is simply these projections (up to the scaling given by singular values). Note that both $\mathbf{U}$ and $\mathbf{V}$ are orthogonal matrices, meaning that principal axes are orthogonal and principal components have correlation zero.

But if you have a linear mixture of some sources, it is usually a non-orthogonal mixture, meaning that truly independent axes (that ICA attempts to find) are (usually) not orthogonal to each other. When you say that

it can be shown that with a rotation, an independent rescaling of each of the rotated axes, and a second rotation you can recover the original, independent axes

you probably mean that taking your data axes (i.e. the set of basis vectors, $\mathbf{I}$), rotating them, scaling and rotating again, you can arrive to the independent axes $\mathbf{A}$. And it is certainly true. If you know $\mathbf{A}$, then you can do SVD of that matrix, and it will give you exactly these rotations and scaling you were talking about. But if you don't know $\mathbf{A}$, then there is no way you can perform an SVD of anything to recover it.

Instead, you need to rely on some assumptions about how the independent components should look like (e.g. they should be as non-Gaussian as possible), and that is what ICA is about.