# Classification on principal components

For my research I am doing classification on the dataset of three variables.

I run unsupervised clustering (based on a histogram peak technique of cluster analysis)and the result I evaluated visually and saw that it was not very good. I tried also supervised classification (minimum distance to means), and unsupervised kmeans (with random seed cluster centroid initialization rule), even worse results. Then I run principal component analysis for these 3 variables, and got almost 99% of the variance in the first component (actually PC1 98.9, PC2 0.97, PC3 0.14). After this I run the classification again but now on the 3 principal components instead of the 3 initial variables. The clustering result was much better, really good. Others did not improve much. And I see that the PC2 despite of its only 1% of the variance is very important for my classification, and PC3 also helps.

And my question is:

What is the statistical explanation of that effect? Maybe, by using 3 orthogonal components as equal variables for classification, I am incresing the influence of small part of information which is in the last principal components? Is it a reasonable way to help classification, or I am doing something crazy from the statistical point of view? And is this effect just a case of good luck and depends on the image, or I can use and recommend this method hereafter?

I saw here questions about using PCA for reducing dimensions before classification, but this is not what I need, I am in opposite interested in the sense of using all (especially last) PCs. Also I saw here the question PCA and random forests, but it was mainly about why classifications improved with additional features and touched my question just at #3 "what if ...", and those answers are not very good for me.

And, my variables are red, green and blue bands of an image. Here is the comparison of the classification results in the most problem spots for land and water. Here are images of PCs and scatterplots of PCs. The scatterplots of the variables:

I am not a statistician and I appreciate not very complicated explanations. Thanks in advance!

• You've got an interesting question. What I strongly suggest you is to give or show your data, for example on 3 scatterplots (v1-v2, v1-v3, v2-v3). Also, tell us please what clustering method and classification method you used and what it was that was not very good with their results. – ttnphns Nov 6 '12 at 5:29
• One preliminary possible explanation of the discrepancy between your clustering on 3 original variables and their 3 PCs might be that you used standardized, not raw, PC scores. That is, the PCs now have equal variances and this weights PC2 and PC3 heavier than PC1. Accidentally, your data splits mostly along PC2. – ttnphns Nov 6 '12 at 5:48
• Thank you @ttnphns for the comments, I added images and plots. And doing classifications I asked for more classes than I needed and then manually reclassified into several I actually want. – nadya Nov 6 '12 at 21:35
• Typically the term "classification" refers to a problem in supervised learning. Here you are using unsupervised learning techniques, so it would be better to use some other word to describe what you are doing. – Bitwise Nov 6 '12 at 23:30
• You are right, just in the image processing area the term "unsupervised classification" is widely used. – nadya Nov 7 '12 at 0:01

Are by chance your scales different?

How do you principal components look like?

Most likely you did not perform appropriate normalization beforehand.

A side effect of doing PCA is that it also will apply z-score normalization to your data set. This gives all three dimensions roughly equal weight. And as you can imagine, using all three attributes and not just one will usually result in better performance.

• Thank you for the answer, I added image of the components and variables. So, my last PCs got high weight and this helped. If PC1 looks like a sum of variables, I am wondering what information goes to PC2 and PC3, or it depends on the image? – nadya Nov 6 '12 at 21:50

One way this could happen is if the data are evenly spaced on the first PC and have a break on the second PC. e.g.

set.seed(2881010)
PC1 <- runif(100, 0, 100)
PC2 <- c(rnorm(50,-2,.5), rnorm(50, 2, .5))
plot(PC1, PC2)

junk <- cbind(PC1, PC2)
library(cluster)
m1 <- diana(junk, diss = FALSE)
m2 <- kmeans(junk, 2)
plot(x = PC1, y = PC2, col = m2\$cluster)


which leads to beautiful clusters on the 2nd dimension, even though it varies much less than the first

• Thank you, I added scatterplots of PCs and variables. Not sure if I have such clusters, I will try to check. – nadya Nov 6 '12 at 23:30

The R,G,B channels are highly correlated and hence its not surprising that you got PC1 98.9. That fact that the PC2 and PC3 are very much influential in classification is not surprising either because of the below -

When we do Principal Component analysis the principal components are direction of maximum variability, then do not guarantee maximum discrimination or separation between classes.

So the 2nd component gives you good classification means data in that direction gives you better discrimination between classes. When you perform Linear Discriminant Analysis(LDA) it gives you the best orthogonal direction components that maximize the inter-class distance and minimize the intra-class distance.

Had you done a LDA may be the 1st component that you would have got would have been closer to PC2. Hope this helps.

• Thank you, that study is finished years ago but your suggestion of LDA is interesting, if I get a similar issue again I will try it. – nadya Mar 2 '17 at 16:57