I have conducted PCA that has reduced the dimensions of my data from more than 20 to 7 (7 PCAs explain about 85% of the total variation). As a second step, I have to cluster my data based on these new 7 PCAs.

My question is: how I should reconstruct/transform my data (cbind/rbind)? As I understand, each PCA is a weighted mix of the original variables. So should I just replace the old variables with the PCAs?

Also, how should I interpret the final results? If there was no PCA, each cluster would incorporate all variables in some proportion. But after PCA, how would I describe each cluster? For instance, cluster 1 contains 40% of PCA 1 and PCA 1, in turn, has xxx loading scores? Something like this? Would greatly appreciate any help on interpreting this.


You are suggesting you are using R. So here is an example for PCA and Kmeans clustering on toy data.

d = mtcars
d2 = prcomp(d, scale=T)
x =d2$x[,1:2]
y = kmeans(x,2)

In this example, the original data has 11 features, and we reduce them into 2 and run kmeans clustering to cluster the data into 2 clusters.

To summarize the code:

  • We have 32 data points (32 cars), and 11 features (car's weight, cylinders etc.), and we convert it into another data matrix X that also has 32 rows, but 2 features.

  • These 2 features are linear combinations of original feature, and do not have a clear physical meaning.

  • When we run clustering, we are still clustering these 32 cars based on new transformed feature.

  • The clustering results shows the cars are similar to each other in transformed feature space. So, if we lose a lot of information in PCA, we cannot say the cars in the same cluster are similar to each other (in original space).

Here is an example to tell what is each cluster: we check the data in one cluster and find the commonalities in original space. For example, In this cars clustering, we can tell the red cluster has the cars that are more heavy, more cylinders and less mpg.

(The clustering is basically a split on PC1, and we can check the loadings to see what is PC1 made of)

enter image description here

  • $\begingroup$ @ Haitao: thanks a lot :) Understood. Just one question: while creating the transformed matrix/data frame, I am replacing the old features (11 in mtcar example) with the 2 new PCAs, such that the transformed frame now has 3 variables, i.e cars and 2 PCAs, correct? And then we cluster on this data frame? $\endgroup$ – Stats Jul 7 '20 at 7:23
  • $\begingroup$ @Stats The transformed data matrix is a 32 by 2, matrix, which means it contains 2 features. the "cars" is actually a row name in R or index in python $\endgroup$ – Haitao Du Jul 7 '20 at 7:25
  • $\begingroup$ perfect..thanks a lot Haitao :) $\endgroup$ – Stats Jul 7 '20 at 7:48

Yes, the PCA loadings components represent the "contribution" of each original variable. These can be positive (same direction) or negative (opposite direction).

Then you perform the clustering using the PCA scores as new variables. You can estimate which of the PCA components contribute more to each cluster in multiple ways (feature importance, statistical test, etc.).

Then you can infer the contribution of the original variables to each cluster, by looking at the weights of the PCA loadings corresponding to components found in the previous step.
This interpretation is quite arbitrary because, as said, the loadings are a combination of all variables with large or small weights. One simple strategy is to rank the weights based on their absolute values and extract the top $N$ variables. Then you can see each loading as approximately dependent on these variables.

  • $\begingroup$ Thanks @Ping: really helpful. My issue is exactly what you have pointed out with clustering in your second paragraph. Each of my clusters is made of all 7 PCs. How should I describe this cluster? Can I simply take the means for each PC across all observations in a cluster? But that does not sound right. Any other method or example I can see, that would be great. Thanks a lot! $\endgroup$ – Stats Jul 9 '20 at 16:32
  • $\begingroup$ You can estimate the feature importance with Random Forest or run a differential test to see which PCA features correlate more with the individual clusters. Then you go back to the original features by looking at the individual loadings. $\endgroup$ – user289381 Jul 9 '20 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.