3
$\begingroup$

When data are a linear mixture of non-gaussian sources, it can be shown that with a rotation, an independent rescaling of each of the rotated axes, and a second rotation you can recover the original, independent axes. Singular value decomposition [SVD] does exactly these three operations -- i.e., it decomposes a matrix X into U, S, and V [i.e, X = USV'], where U & V are rotation matrices. SVD even finds that first rotation [i.e., the PCs] and the scaling for axes [the eigenvalues of the covariance matrix], but the second rotation isn't the rotation to the independent axes. I know the dimensions of the second rotation are wrong, but still...

So here's my [possibly very naive] question: why doesn't SVD find this second rotation to recover the original, independent axes?

There's been a lot posted on how SVD relates to PCA, which makes sense -- but how does SVD relate to ICA? Why can't you recover the unmixing matrix with SVD? or is this just the wrong question to ask...

EDIT:

After reading about this a bit more, I feel like I have a good intuition for why SVD, despite performing the necessary matrix operations [rotate, rescale, rotate], doesn't find the independent components. Both rotation matrices are only sensitive to the correlations in the data -- the U & V matrices are, respectively, the eigenvectors of XX' and X'X. Thus, these two rotations are only sensitive to second-order correlations in the data -- they are not sensitive to higher-order moments [e.g., kurtosis], and these higher-order moments are necessary to find independent projections.

This seems satisfying to me. If others have insights, please let me know!

$\endgroup$
3
$\begingroup$

I think you are (or at least were) confused about several things here.

Doing SVD of your data matrix $\mathbf{X}$ amounts to doing the PCA. Decomposition $\mathbf{X}=\mathbf{USV^\top}$ decomposes the data into principal axes (eigenvectors of the covariance matrix) and principal "scores", or "principal components", i.e. projections of the data onto the principal axes. The "other rotation matrix" you were asking about is simply these projections (up to the scaling given by singular values). Note that both $\mathbf{U}$ and $\mathbf{V}$ are orthogonal matrices, meaning that principal axes are orthogonal and principal components have correlation zero.

But if you have a linear mixture of some sources, it is usually a non-orthogonal mixture, meaning that truly independent axes (that ICA attempts to find) are (usually) not orthogonal to each other. When you say that

it can be shown that with a rotation, an independent rescaling of each of the rotated axes, and a second rotation you can recover the original, independent axes

you probably mean that taking your data axes (i.e. the set of basis vectors, $\mathbf{I}$), rotating them, scaling and rotating again, you can arrive to the independent axes $\mathbf{A}$. And it is certainly true. If you know $\mathbf{A}$, then you can do SVD of that matrix, and it will give you exactly these rotations and scaling you were talking about. But if you don't know $\mathbf{A}$, then there is no way you can perform an SVD of anything to recover it.

Instead, you need to rely on some assumptions about how the independent components should look like (e.g. they should be as non-Gaussian as possible), and that is what ICA is about.

$\endgroup$
  • $\begingroup$ thanks & yup i was originally confused but SVD is only sensitive to second-order correlations, exactly as you point out. and the point about linear mixtures are usually non-orthogonal is right on. $\endgroup$ – user42164 Mar 21 '14 at 16:33
  • $\begingroup$ orthogonal matrices, meaning that principal axes are orthogonal and principal components have correlation zero These are two different things. Orthogonal (orthonormal) matrix says of the orthogonal rotation of the data. This itself doesn't automatically mean that correlation is removed. Correlation is removed by that eigenvalues are successively maximized. $\endgroup$ – ttnphns Apr 27 '14 at 3:06
  • 1
    $\begingroup$ @ttnphns: Yes, these are two different things. But note that I was talking about two matrices: U and V. If X is the data matrix with variables in columns, then SVD decomposition $X=USV^\top$ gives principal axes (eigenvectors of the covariance matrix) in V and loadings (up to the scaling) in U. Both matrices have orthonormal columns. In case of V it means that principal axes are orthogonal, and in case of U it means that principal components have correlation zero. Hope it makes sense. $\endgroup$ – amoeba Apr 27 '14 at 21:07
  • $\begingroup$ "they should be as non-Gaussian as possible" What does this mean exactly? $\endgroup$ – thecity2 Aug 21 '18 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.