2
$\begingroup$

I know the generally reasons of using correlation matrix vs a covariance matrix when doing PCA (and visa versa) however when thinking about the eigenvectors (principal components of the data) of each of these matrices I still have a question which is:

Obviously, these eigenvectors will be different however since we are just scaling one to get the other is there just a 1:1 mapping to the eigenvectors of a correlation matrix to a covariance matrix? I.e are the principal components we get for one conceptually the same just in different coordinate system or are there total different interpretations for each set of principal components we get when doing PCA with correlation or covariance?

$\endgroup$
  • 1
    $\begingroup$ My answer at stats.stackexchange.com/questions/140711 offers illustrations that may help give you the intuition to answer your questions easily. $\endgroup$ – whuber Feb 13 at 22:40
  • $\begingroup$ @whuber appreciate the comment. Helpful post indeed. So, am I correct in saying that then the principal components are different when using correlation or covariance matrices? Because we change the scale, and per your post this does have an impact (as it changed the # of clusters in your case) thus - depending on the data the principal component gleaned from PCA via correlation or PCA via covariance maybe more useful than the other? $\endgroup$ – hhprogram Feb 14 at 19:41
1
$\begingroup$

The starting covariance and correlation matrices are a different coordination system and only under very limited circumstance would they bear close resemblance.

In a covariance matrix the variables can have any magnitude of relationship to each other, but in correlation it is constrained so the total variance is 1 and the covariance any fraction of that.

I work mainly with digital signals and they provide evidence that there is no guarantee of correspondence. In the signals I work with the noise level is inversely related to the square root of the intensity. This means for variables with high intensity the covariance matrix picks these out strongly over weak variables. In a correlation matrix the weak variables are given the same weighting even though they are more noisy. Thus a correlation matrix increases noise.

Noise by definition is orthogonal to signal, so it will not be possible to have 1 to 1 correspondence if you have sizeable differences in variable intensity within a signal. If the noise after standardisation is greater than the signal, then your signal disappears from your eigenvectors (or to be precise, it gets split across many noisy eigenvectors based on chance Cross correlations specific to your training set)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.