In PCA, does it make a difference if we pick principal components of the inverse covariance matrix OR if we drop eigenvectors of the covariance matrix corresponding to large eigenvalues?
This is related to the discussion in in this post.
$\begingroup$
$\endgroup$
0
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
Observe that for positive definite covariance matrix $\mathbf \Sigma = \mathbf{UDU}'$ the precision is $\boldsymbol \Sigma^{-1} = \mathbf {U D}^{-1} \mathbf U'$.
So the eigenvectors stay the same, but the eigenvalues of the precision are the reciprocals of the eigenvalues of the covariance. That means the biggest eigenvalues of the covariance will be the smallest eigenvalues of the precision. As you have the inverse, positive definiteness guarantees all eigenvalues are greater than zero.
Hence if you retain the eigenvectors relating to the $k$ smallest eigenvalues of the precision this corresponds to ordinary PCA. Since we have already taken reciprocals ($\bf D^{-1}$), only the square root of the precision eigenvalues should be used in order to complete the whitening of the transformed data.
-
$\begingroup$ +1 but I think your "So yes, it makes a difference" sentence might be confusing for the OP; the Q is not very clear but I think they were asking whether there is a difference between selecting largest eigenvalues of inv cov matrix and selecting smallest eigenvalues ( = dropping the largest ones) of the cov matrix. To this question the answer is that it is equivalent. So perhaps if you simply cut out this sentence, the answer will be clearer. $\endgroup$– amoebaCommented Mar 31, 2016 at 20:31
-
$\begingroup$ Thanks, I see what you mean and have edited accordingly. $\endgroup$ Commented Mar 31, 2016 at 20:34
-
$\begingroup$ Actually the last sentence was good, I would have kept it! $\endgroup$– amoebaCommented Mar 31, 2016 at 20:37
-
$\begingroup$ @conjectures Thank you, that's the perfect explanation. $\endgroup$ Commented Apr 6, 2016 at 5:52
$\begingroup$
$\endgroup$
1
In addition, the inverse covariance matrix is proportionnal to the partial correlation between the vectors:
Corr(Xi, Xj | (Xothers )
Correlation between Xi and Xj when all others are fixed, it is very useful for time series.
-
1$\begingroup$ This is true, but what does that have to do with PCA? $\endgroup$– amoebaCommented Mar 31, 2016 at 20:23