Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

What is a good metric for assessing the quality of a pca? I performed this algorithm on a dataset. My objective was to reduce the number of features (the information was very redundant). I know the percentage of variance kept is a good indicator of how much information we keep, be are there other information metrics I can use to make sure I removed redundant information and didn't 'lose' such information?

share|improve this question
2  
Strictly speaking, there is no "redundant" information, unless your initial data were perfectly collinear. One usually sees percentage of variance retained ("we used the first five principal components, which accounted for 90% of the variance"). I'm interested in seeing alternatives. –  Stephan Kolassa May 27 at 7:47
    
Since one of your tags is info theory: An indirect way of assessing whether PCA works is to check the assumptions under which information theory tells us it has low info loss for a given dimension reduction. Wiki says this is so when your data is a sum of gaussian signal plus gaussian noise. en.wikipedia.org/wiki/… –  CloseToC May 27 at 15:56

2 Answers 2

up vote 13 down vote accepted

I assume part of this question is whether other metric exist besides the cumulative percent variance (CPV) and the similar scree plot approach. The answer to this is, yes, many.

A great paper on some options is Valle 1999:

Selection of the Number of Principal Components:  The Variance of the Reconstruction Error Criterion with a Comparison to Other Methods† Sergio Valle,Weihua Li, and, and S. Joe Qin* Industrial & Engineering Chemistry Research 1999 38 (11), 4389-4401

It goes over CPV, but also Parallel Analysis, Cross-validation, Variance of the reconstruction error (VRE), information criteria based methods, and more. You might follow the recommendation made by the paper after comparing and use the VRE, but cross-validation based on PRESS also works well in my experience and they get good results with that too. In my experience, CPV is convenient and easy, and does a decent job, but those two methods are usually better.

There are other ways to evaluate how good your PCA model is if you know more about the data. One way is to compare the estimated PCA loadings to the true ones if you know them (which you would in simulations). This can be done by calculating the bias of the estimated loadings to the true ones. The bigger your bias, the worse your model. For how to do that, you can check out this paper where they use this approach to compare methods. It is not usable in real data cases though, where you don't know the true PCA loadings. This speaks less to how many components you removed, than to the bias of your model due to the influence of outlying observations, but it still serves as a model quality metric.

share|improve this answer
4  
Link to Valle, Li and Qin paper –  Zhubarb May 27 at 12:22
    
@Zhubarb: thanks for adding a link to the paper. –  Deathkill14 May 27 at 13:31

There are also measures based on information-theoretic criteria like

Rissanen's MDL (and variations)

share|improve this answer
    
@user:45382 Yes, that is another one. It is also touched upon in the paper Zhubarb links to. –  Deathkill14 May 27 at 16:00
    
@Deathkill14 correct i read the paper, information-theoretic measures are mentioned (in fact as good alternatives) –  Nikos M. May 27 at 16:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.