9
$\begingroup$

I ran PCA on 17 quantitative variables in order to obtain a smaller set of variables, that is principal components, to be used in supervised machine learning for classifying instances into two classes. After PCA the PC1 accounts for 31% of the variance in the data, PC2 accounts for 17%, PC3 accounts for 10%, PC4 accounts for 8%, PC5 accounts for 7% and PC6 accounts for 6%.

However, when I look at mean differences between PCs among the two classes, surprisingly, PC1 is not a good discriminator between the two classes. Remaining PCs are good discriminators. In addition, PC1 becomes irrelevant when used in a decision tree which means that after tree pruning it is not even present in the tree. The tree consists of PC2-PC6.

Is there any explanation for this phenomenon? Can it be something wrong with the derived variables?

$\endgroup$
  • 4
    $\begingroup$ Read this recent question stats.stackexchange.com/q/79968/3277 with further link in it. Since PCA does not know about the existence of the classes it does not guarantee that any of the PCs will be really good discriminators; all the more that PC1 will be a good discriminator. See also two pictures as example here. $\endgroup$ – ttnphns Dec 24 '13 at 0:26
  • 1
    $\begingroup$ See also What can cause PCA to worsen results of a classifier?, in particular the figures in the answer by @vqv. $\endgroup$ – amoeba Jan 14 '15 at 22:39
13
$\begingroup$

It can also happen if the variables are not scaled to have unit variance before doing PCA. For example, for these data (notice that the $y$ scale only goes from $-0.5$ to $1$ whereas $x$ goes from $-3$ to $3$):

enter image description here

PC1 is approximately $x$ and accounts for almost all the variance, but has no discriminatory power, whereas PC2 is $y$ and discriminates perfectly between the classes.

$\endgroup$
  • $\begingroup$ Hi, thanks for your answer! How the scaling should be done? (x-mean)/sd? $\endgroup$ – Frida Dec 24 '13 at 0:33
  • $\begingroup$ Yes, in R I used prcomp(x, center=T, scale=T) which is the same as doing (x-mean)/sd. In this example, you would find that neither principal component is a good discriminator between the classes; it only works if they are both used together. $\endgroup$ – Flounderer Dec 24 '13 at 0:35
  • $\begingroup$ I understand that the PCs are complementary but is there any explanation behind this? To be honest, this is the first time for me to obtain a weaker PC1 as compared to others. $\endgroup$ – Frida Dec 24 '13 at 0:38
  • $\begingroup$ @Frida: ttnphns' comment is on-target. See en.wikipedia.org/wiki/Linear_discriminant_analysis third paragraph. In some sense it's luck that PCA turns out to be useful so often. $\endgroup$ – Wayne Dec 24 '13 at 1:43
5
$\begingroup$

I assume that the answer and the example provided by @Flounderer imply this, but I think it worth spelling this out. Principle component analysis is label (classification) indifferent. All it does is to transform some high dimensional data to another dimensional space. It might help in classification attempts by, for example, creating data set that is easier separable by a particular method. However, this is only a by-product (or a side effect) of PCA.

$\endgroup$
1
$\begingroup$

When we do Principal Component analysis the principal components are direction of maximum variability, then do not guarantee maximum discrimination or separation between classes.

So the 2nd component gives you good classification means data in that direction gives you better discrimination between classes. When you perform Linear Discriminant Analysis(LDA) it gives you the best orthogonal direction components that maximize the inter-class distance and minimize the intra-class distance.

So if you do LDA on the data instead of PCA probably one of the very fist components would be closer to PC6 than to PC1. Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.