Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

PCA reduces data set dimensions while trying to keep most variations in data set.

PCA can be used as a dimension reducing technique in discrimination, however it tries to keep the most discrimination power rather than keep most variations as more variation does not necessarily mean the separation of different group of data is maximum. So Chang (1983) in his paper "On using principal components before separating a mixture of two multivariate normal distributions" suggest using Mahalanobis distance in some way to measure it. His paper can be found here:

Does the method work well when the data size are very different (consider two groups only), Say sample sizes have ratio 1:99? If it does not work well, how to handle the problem?

I have read literature about rebalancing data set for PCA, but there are no theoretical and mathematical explanation for how rebalance data can help increase the performance of PCA. Is there any literature for the mathematical explanation part?

share|improve this question
Let me ask you to clarify and detail your question. First, what is "imbalance"? Groups of very different n or else? So, do you have your sample split into groups or not? Second, PCA (?)tries to keep the most discrimination power rather than keep most variations seems to contradict with PCA trying to keep most variations in data set. Please, elucidate. – ttnphns Jul 7 '12 at 9:28
Thanks. I have edited it. The question should be more clearly now. – zca0 Jul 7 '12 at 10:00
Is the essence of your question this: There's a multivariate data presumably heterogeneous due to presence of clusters (groups) in it; the clusters are not known, so discriminant analysis is not applicable. Is it then possible to use PCA in place of it, to differentiate the groups? If yes, how to make PCA maximally effective for this discriminative task? Etc. What I recommend you is to rework this and the other (just next) your questions and merge them into one question. – ttnphns Jul 7 '12 at 10:43
up vote 1 down vote accepted

I addressed your confusion about PCA in your other question. The correct version of your statement is that PCA attempts to characterize most of the variation in the in a lower dimensional space. As I explain in the answer to the other post PCA is not used for discrimination. It has been used to do clustering after reducing dimension to a subset of the principal components.

If by imbalance of the data you mean one cluster center has many more points arund it than another then that may affect the ability of any clustering method at identifying the smaller cluster but the magnitude of separation of the clusters is probably more important.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.