12
$\begingroup$

What is a good metric for assessing the quality of principal component analysis (PCA)?

I performed this algorithm on a dataset. My objective was to reduce the number of features (the information was very redundant). I know the percentage of variance kept is a good indicator of how much information we keep, be are there other information metrics I can use to make sure I removed redundant information and didn't 'lose' such information?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Strictly speaking, there is no "redundant" information, unless your initial data were perfectly collinear. One usually sees percentage of variance retained ("we used the first five principal components, which accounted for 90% of the variance"). I'm interested in seeing alternatives. $\endgroup$
    – Stephan Kolassa
    Commented May 27, 2014 at 7:47
  • $\begingroup$ Since one of your tags is info theory: An indirect way of assessing whether PCA works is to check the assumptions under which information theory tells us it has low info loss for a given dimension reduction. Wiki says this is so when your data is a sum of gaussian signal plus gaussian noise. en.wikipedia.org/wiki/… $\endgroup$
    – CloseToC
    Commented May 27, 2014 at 15:56

2 Answers 2

Reset to default
18
$\begingroup$

I assume part of this question is whether other metric exist besides the cumulative percent variance (CPV) and the similar scree plot approach. The answer to this is, yes, many.

A great paper on some options is Valle 1999:

It goes over CPV, but also Parallel Analysis, Cross-validation, Variance of the reconstruction error (VRE), information criteria based methods, and more. You might follow the recommendation made by the paper after comparing and use the VRE, but cross-validation based on PRESS also works well in my experience and they get good results with that too. In my experience, CPV is convenient and easy, and does a decent job, but those two methods are usually better.

There are other ways to evaluate how good your PCA model is if you know more about the data. One way is to compare the estimated PCA loadings to the true ones if you know them (which you would in simulations). This can be done by calculating the bias of the estimated loadings to the true ones. The bigger your bias, the worse your model. For how to do that, you can check out this paper where they use this approach to compare methods. It is not usable in real data cases though, where you don't know the true PCA loadings. This speaks less to how many components you removed, than to the bias of your model due to the influence of outlying observations, but it still serves as a model quality metric.

Improve this answer
$\endgroup$
1
3
$\begingroup$

There are also measures based on information-theoretic criteria like

Rissanen's MDL (and variations)

Improve this answer
$\endgroup$
3
  • $\begingroup$ @user:45382 Yes, that is another one. It is also touched upon in the paper Zhubarb links to. $\endgroup$
    – Deathkill14
    Commented May 27, 2014 at 16:00
  • $\begingroup$ @Deathkill14 correct i read the paper, information-theoretic measures are mentioned (in fact as good alternatives) $\endgroup$
    – Nikos M.
    Commented May 27, 2014 at 16:06
  • $\begingroup$ An excellent theoretical paper on MDL, MML and Bayesianism: Vitany & Li, Ideal MDL and Its Relation To Bayesianism citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.56.8580 . Also other model selection methods such as AIC and BIC are effectively implementations of MDL. $\endgroup$
    – ggll
    Commented Oct 11, 2018 at 8:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.