# Can I use PCA to do variable selection for cluster analysis?

I have to reduce the number of variables to conduct a cluster analysis. My variables are strongly correlated, so I thought to do a Factor Analysis PCA (principal component analysis). However, if I use the resulting scores, my clusters are not quite correct (compared to previous classifications in literature).

Question:

Can I use the rotation matrix to select the variables with the biggest loads for each component/factor and use only these variables for my clustering?

Any bibliographic references would also be helpful.

Update:

Some clarifiations:

• My goal: I have to run a clusters analysis with two-step algorithm by SPSS, but my variables are not independents, so I thought about discarding some of them.

• My dataset: I am working on 15 scalar parameters (my variables) of 100,000 cases. Some variables are strongly correlated ($>0.9$ Pearson)

• My doubt: Since I need only independent variables, I thought to run a Principal Component Analysis (sorry: I wrongly talked about Factor Analysis in my original question, my mistake) and select only the variables with the biggest loadings for each component. I know that the PCA process presents some arbitrary steps, but I found out that this selection is actually similar to the "method B4" proposed by I.T. Jolliffe (1972 & 2002) to select variables and suggested also by J.R. King & D.A. Jackson in 1999.

So I was thinking to select in this way some sub-groups of independent variables. I will then use the groups to run different cluster analysis and I will compare the results.

• If you know the correct answer, why do the analysis at all? – StasK Oct 14 '11 at 1:13
• On another note, why do you think you need to reduce the number of variables for cluster analysis? I don't think any of the modern tools of cluster analysis have any limitations as to the number of input variables. Of course if you have a test with 120 items, things will get complicated with it. – StasK Oct 14 '11 at 1:18
• possible duplicate of Using principal component analysis (PCA) for feature selection – amoeba says Reinstate Monica Feb 5 '15 at 15:43
• It seems to me that the addition of the cluster analysis aspect of this Q makes it distinct enough to remain open. – gung - Reinstate Monica Feb 5 '15 at 16:11
• You seem to apply stricter criteria to duplicates than me, @gung; perhaps you are right (and the voting does not go well on this one either). However, in this particular case the OP was asking about the simplest PCA-based feature selection (as clarified in his update) that is covered in the thread I suggested. On the other hand, StasK posted here an interesting answer that is specifically about clustering... – amoeba says Reinstate Monica Feb 5 '15 at 21:02

I will, as is my custom, take a step back and ask what it is you are trying to do, exactly. Factor analysis is designed to find latent variables. If you want to find latent variables and cluster them, then what you are doing is correct. But you say you simply want to reduce the number of variables - that suggests principal component analysis, instead.

However, with either of those, you have to interpret cluster analysis on new variables, and those new variables are simply weighted sums of the old ones.

How many variables have you got? How correlated are they? If there are far too many, and they are very strongly correlated, then you could look for all correlations over some very high number, and randomly delete one variable from each pair. This reduces the number of variables and leaves the variables as they are.

Let me also echo @StasK about the need to do this at all, and @rolando2 about the usefulness of finding something different from what has been found before. As my favorite professor in grad school used to say "If you're not surprised, you haven't learned anything".

• first of all, I am sorry: I am actually referring to a Principal components analysis, not to factor analysis, my mistake. Moreover, I was looking to find a way to not select arbitrarily witch correlated variable I shall keep. I add more info about the problem above.. thank you again – en. Nov 12 '11 at 18:40

A way to perform factor analysis and cluster analysis at the same time is through structural equation mixture models. In these models, you postulate that there are separate models (in this case, factor models) for each cluster. You would need to have the mean analysis along with the covariance analysis, and be concerned with identification to a greater extent that in plain vanilla factor analysis. The idea approached from SEM side appears in Jedidi et. al. (1997), and from clustering side, in model-based clustering by Adrian Raftery. This type of analysis is, apparently, available in Mplus.

• thank you for the inputs, specially for the references, but I wrongly refered to Factor Analysis: I was actually thinking about Principal Components in order to reduce my variables set to a sub-group of independent variables. my mistake – en. Nov 12 '11 at 18:50

I don't think it's a matter of "correctness" pure and simple, but rather whether it will accomplish what you are looking to do. The approach you describe will end up clustering according to certain factors, in a watered-down way, since you will be using only one indicator to represent each factor. Each such indicator figures to be an imperfect stand-in for the underlying, latent factor. That's one issue.

Another issue is that factor analysis itself, as I (and many other people) have recounted, is full of subjective decisions involving how to deal with missing data, number of factors to extract, how to extract, whether and how to rotate, and so on. So it may be far from clear that the factors you may have extracted in a quick, software-default manner (as I think you have implied) are the "best" in any sense.

Altogether, then, you may have used watered-down versions of factors that are themselves debatable as being the best ways to characterize the themes underlying your data. I wouldn't expect that the clusters resulting from such input variables would be the most informative or the most distinct.

On another note, it seems interesting that you consider it a problem to have cluster memberships/profiles that don't line up with what other researchers have found. Sometimes disconfirming findings can be very healthy!

• thank you very much, I have added more information above to specify my doubts – en. Nov 12 '11 at 18:46

What could be happening in your case is that the factors extracted in Factor Analysis have compensating positive and negative loads from the original variables. This would diminish differentiability that is the purpose of clustering.