In PCA, does it make a difference if we pick principal components of the inverse covariance matrix OR if we drop eigenvectors of the covariance matrix corresponding to large eigenvalues?

This is related to the discussion in in this post.


Observe that for positive definite covariance matrix $\mathbf \Sigma = \mathbf{UDU}'$ the precision is $\boldsymbol \Sigma^{-1} = \mathbf {U D}^{-1} \mathbf U'$.

So the eigenvectors stay the same, but the eigenvalues of the precision are the reciprocals of the eigenvalues of the covariance. That means the biggest eigenvalues of the covariance will be the smallest eigenvalues of the precision. As you have the inverse, positive definiteness guarantees all eigenvalues are greater than zero.

Hence if you retain the eigenvectors relating to the $k$ smallest eigenvalues of the precision this corresponds to ordinary PCA. Since we have already taken reciprocals ($\bf D^{-1}$), only the square root of the precision eigenvalues should be used in order to complete the whitening of the transformed data.

  • $\begingroup$ +1 but I think your "So yes, it makes a difference" sentence might be confusing for the OP; the Q is not very clear but I think they were asking whether there is a difference between selecting largest eigenvalues of inv cov matrix and selecting smallest eigenvalues ( = dropping the largest ones) of the cov matrix. To this question the answer is that it is equivalent. So perhaps if you simply cut out this sentence, the answer will be clearer. $\endgroup$ – amoeba says Reinstate Monica Mar 31 '16 at 20:31
  • $\begingroup$ Thanks, I see what you mean and have edited accordingly. $\endgroup$ – conjectures Mar 31 '16 at 20:34
  • $\begingroup$ Actually the last sentence was good, I would have kept it! $\endgroup$ – amoeba says Reinstate Monica Mar 31 '16 at 20:37
  • $\begingroup$ @conjectures Thank you, that's the perfect explanation. $\endgroup$ – Mustafa Arif Apr 6 '16 at 5:52

In addition, the inverse covariance matrix is proportionnal to the partial correlation between the vectors:

Corr(Xi, Xj | (Xothers )

Correlation between Xi and Xj when all others are fixed, it is very useful for time series.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.