Can I use PCA to do variable selection for cluster analysis?

Question

I have to reduce the number of variables to conduct a cluster analysis. My variables are strongly correlated, so I thought to do a Factor Analysis PCA (principal component analysis). However, if I use the resulting scores, my clusters are not quite correct (compared to previous classifications in literature).

Question:

Can I use the rotation matrix to select the variables with the biggest loads for each component/factor and use only these variables for my clustering?

Any bibliographic references would also be helpful.

Update:

Some clarifiations:

• My goal: I have to run a clusters analysis with two-step algorithm by SPSS, but my variables are not independents, so I thought about discarding some of them.

• My dataset: I am working on 15 scalar parameters (my variables) of 100,000 cases. Some variables are strongly correlated ($>0.9$ Pearson)

• My doubt: Since I need only independent variables, I thought to run a Principal Component Analysis (sorry: I wrongly talked about Factor Analysis in my original question, my mistake) and select only the variables with the biggest loadings for each component. I know that the PCA process presents some arbitrary steps, but I found out that this selection is actually similar to the "method B4" proposed by I.T. Jolliffe (1972 & 2002) to select variables and suggested also by J.R. King & D.A. Jackson in 1999.

  So I was thinking to select in this way some sub-groups of independent variables. I will then use the groups to run different cluster analysis and I will compare the results.

Answer 1

I will, as is my custom, take a step back and ask what it is you are trying to do, exactly. Factor analysis is designed to find latent variables. If you want to find latent variables and cluster them, then what you are doing is correct. But you say you simply want to reduce the number of variables - that suggests principal component analysis, instead.

However, with either of those, you have to interpret cluster analysis on new variables, and those new variables are simply weighted sums of the old ones.

How many variables have you got? How correlated are they? If there are far too many, and they are very strongly correlated, then you could look for all correlations over some very high number, and randomly delete one variable from each pair. This reduces the number of variables and leaves the variables as they are.

Let me also echo @StasK about the need to do this at all, and @rolando2 about the usefulness of finding something different from what has been found before. As my favorite professor in grad school used to say "If you're not surprised, you haven't learned anything".

Answer 2

A way to perform factor analysis and cluster analysis at the same time is through structural equation mixture models. In these models, you postulate that there are separate models (in this case, factor models) for each cluster. You would need to have the mean analysis along with the covariance analysis, and be concerned with identification to a greater extent that in plain vanilla factor analysis. The idea approached from SEM side appears in Jedidi et. al. (1997), and from clustering side, in model-based clustering by Adrian Raftery. This type of analysis is, apparently, available in Mplus.

Answer 3

I don't think it's a matter of "correctness" pure and simple, but rather whether it will accomplish what you are looking to do. The approach you describe will end up clustering according to certain factors, in a watered-down way, since you will be using only one indicator to represent each factor. Each such indicator figures to be an imperfect stand-in for the underlying, latent factor. That's one issue.

Another issue is that factor analysis itself, as I (and many other people) have recounted, is full of subjective decisions involving how to deal with missing data, number of factors to extract, how to extract, whether and how to rotate, and so on. So it may be far from clear that the factors you may have extracted in a quick, software-default manner (as I think you have implied) are the "best" in any sense.

Altogether, then, you may have used watered-down versions of factors that are themselves debatable as being the best ways to characterize the themes underlying your data. I wouldn't expect that the clusters resulting from such input variables would be the most informative or the most distinct.

On another note, it seems interesting that you consider it a problem to have cluster memberships/profiles that don't line up with what other researchers have found. Sometimes disconfirming findings can be very healthy!

Answer 4

What could be happening in your case is that the factors extracted in Factor Analysis have compensating positive and negative loads from the original variables. This would diminish differentiability that is the purpose of clustering.

Can you break up each extracted factor into 2 - one having just the positive loadings, the other just the negative loadings?

Replace the factor scores for each case for each factor by positive scores and negative scores and try clustering on this new set of scores.

Please drop in a line if this works for you.

Answer 5

You could scan both for high values and also for low values and leave all variables in the factors. This way, there is no need to cut up the factors. If you split Factor 1 (say) a certain way based on the signs of the loadings, in Factor 2, the signs may be quite different. Would you then cut up Factor 2 differently from Factor 1? This seems to be confusing.