1
$\begingroup$

I have a high dimensional dataset (20k rows, 50 variables). When running PCA on it, PC1 is about 20%, PC2 is 16%, etc. I have to go up to PC30 to get 90% explained variance.

What I'm trying to do is to perform PCA on my data, do k-means clustering, and be able to say PC1 is mostly "gender" related, PC2 is mostly about "income", etc, using the eigenvectors (from my research online, that's how to interpret PCs?).

So what I'm getting is this: low PC1 and PC2s. When I look at the PC1 eigenvalues, the highest value is like .2. When I look at something like PC 24 (which has an extremely low variance explained), I do see one eigenvalue that is .6.

So in terms of interpretability, I'm not sure how to proceed.

  1. Can I rely on a low eigenvalue on PC1 which has .2 variance explained?
  2. Can I rely on a fairly high eigenvalue, but with a PC that has a very very low variance explained?
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ As mkt already pointed, it is likely - but not guaranteed - that clusters, if they exist, will show up in the first few (say, one, two, or three) PCs. Imagine a simple example with 2D data, so 2 PCs in all; and the clusters are much oblong and close to each other and also parallel. PC1 will not uncover the clusters, but the weaker PC2 will. Fortunately, we do not often have such such data. For example, in social sciences clusters are typically round or weakly oblong, and we can hope that PCA extracting few first PCs will detect the clusters, if any. $\endgroup$
    – ttnphns
    Sep 27, 2022 at 14:45
  • 1
    $\begingroup$ Try to consider also the domain called "assessing cluster tendency" - evaluation whether there are clusters at all in the data. Some of such probing methods can be done on raw data, before PCA. $\endgroup$
    – ttnphns
    Sep 27, 2022 at 14:49

2 Answers 2

Reset to default
3
$\begingroup$

The title of this thread and the body point towards different questions. I'll address the title question here.

There is no guarantee that the principal components (PCs) that capture most of the variance in the data are the same as those that are informative for any purpose, including clustering. This depends on the nature of the variation in the data. Given that we have chosen the variables to summarise with a PCA, it is unlikely but possible for the first PC to be uninformative, and the 10th (or 20th, or nth) PC to capture useful information. Visual examination of patterns in each PC and an examination of the loadings are helpful steps to establish whether they are informative.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I would also add that the "optimal" clustering may depend on what we intend to do with the clusters. Or in other words, the most interesting/relevant variation in the data can change with the problem. $\endgroup$
    – dipetkov
    Sep 27, 2022 at 14:46
  • $\begingroup$ @dipetkov Excellent point! $\endgroup$
    – mkt
    Sep 27, 2022 at 14:47
0
$\begingroup$

PCA is a good technique for data dimension reduction. Based on what you shared, few thoughts as suggestions:

  1. Eigen values measure the variability for each component. You should choose the components with eigen values greater or equal to one.

  2. The sum of all eigen values produces a measure of variability in your data. If your eigen values are close to zero, variability is to low and using only two components as input for clustering will not produce a robust scheme for groups.

  3. What to do in this case. You can scale your data ($(x_j-mean_j)/\sigma_j$). Apply that for all your variables and the results must change. In case you decided not scaling, your should take all components that accumulate at least 90% of variability in data, in this situation as you mentioned 30 components.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.