13
$\begingroup$

I ran PCA on 17 quantitative variables in order to obtain a smaller set of variables, that is principal components, to be used in supervised machine learning for classifying instances into two classes. After PCA the PC1 accounts for 31% of the variance in the data, PC2 accounts for 17%, PC3 accounts for 10%, PC4 accounts for 8%, PC5 accounts for 7% and PC6 accounts for 6%.

However, when I look at mean differences between PCs among the two classes, surprisingly, PC1 is not a good discriminator between the two classes. Remaining PCs are good discriminators. In addition, PC1 becomes irrelevant when used in a decision tree which means that after tree pruning it is not even present in the tree. The tree consists of PC2-PC6.

Is there any explanation for this phenomenon? Can it be something wrong with the derived variables?

Improve this question
$\endgroup$
2
  • 5
    $\begingroup$ Read this recent question stats.stackexchange.com/q/79968/3277 with further link in it. Since PCA does not know about the existence of the classes it does not guarantee that any of the PCs will be really good discriminators; all the more that PC1 will be a good discriminator. See also two pictures as example here. $\endgroup$
    – ttnphns
    Commented Dec 24, 2013 at 0:26
  • 2
    $\begingroup$ See also What can cause PCA to worsen results of a classifier?, in particular the figures in the answer by @vqv. $\endgroup$
    – amoeba
    Commented Jan 14, 2015 at 22:39

3 Answers 3

Reset to default
15
$\begingroup$

It can also happen if the variables are not scaled to have unit variance before doing PCA. For example, for these data (notice that the $y$ scale only goes from $-0.5$ to $1$ whereas $x$ goes from $-3$ to $3$):

enter image description here

PC1 is approximately $x$ and accounts for almost all the variance, but has no discriminatory power, whereas PC2 is $y$ and discriminates perfectly between the classes.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi, thanks for your answer! How the scaling should be done? (x-mean)/sd? $\endgroup$
    – Frida
    Commented Dec 24, 2013 at 0:33
  • $\begingroup$ Yes, in R I used prcomp(x, center=T, scale=T) which is the same as doing (x-mean)/sd. In this example, you would find that neither principal component is a good discriminator between the classes; it only works if they are both used together. $\endgroup$
    – Flounderer
    Commented Dec 24, 2013 at 0:35
  • $\begingroup$ I understand that the PCs are complementary but is there any explanation behind this? To be honest, this is the first time for me to obtain a weaker PC1 as compared to others. $\endgroup$
    – Frida
    Commented Dec 24, 2013 at 0:38
  • $\begingroup$ @Frida: ttnphns' comment is on-target. See en.wikipedia.org/wiki/Linear_discriminant_analysis third paragraph. In some sense it's luck that PCA turns out to be useful so often. $\endgroup$
    – Wayne
    Commented Dec 24, 2013 at 1:43
6
$\begingroup$

I assume that the answer and the example provided by @Flounderer imply this, but I think it worth spelling this out. Principal component analysis (PCA) is label (classification) indifferent. All it does is to transform some high dimensional data to another dimensional space. It might help in classification attempts by, for example, creating data set that is easier separable by a particular method. However, this is only a by-product (or a side effect) of PCA.

Improve this answer
$\endgroup$
1
$\begingroup$

When we do Principal Component analysis the principal components correspond to the directions of maximum variability, they do not guarantee maximum discrimination or separation between classes.

So the 2nd component gives you good classification means data in that direction gives you better discrimination between classes. When you perform Linear Discriminant Analysis(LDA) it gives you the best orthogonal direction components that maximize the inter-class distance and minimize the intra-class distance.

So if you do LDA on the data instead of PCA probably one of the very first components would be closer to PC6 than to PC1. Hope this helps.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.