Can PCA allow to identify redundant variables that can be removed before doing cluster analysis?

Question

I hope this is suitable for this forum: I am new to PCA and what I ultimately want to do is perform cluster analysis on my dataset.

I have 20 physical descriptor variables for organisms, each with 300 datapoints. I produced a correlation matrix to look at which variables may be correlated with each other, wherein I found there were a number of variables that were correlated with each other.

I want to remove any redundant variables from the analysis (ones that aren't really contributing anything) before I carry out a cluster analysis on my dataset. I carried out PCA and found that 3 principal components account for about 90% of the variance of my data. My question pertains to how I interpret this output: Do i need to identify what variables were included in these three principal components, and then remove the variables that were not included? Is this even the correct approach to identifying variables that are not contributing any information to the dataset?

For context: What I ultimately want to do is to reduce the number of variables required to describe groups of organisms, which will allow me to model other organisms that share these physical descriptors (but for which there are no data collected).

---

Edit

Thanks for the advice everyone. @hssay: you have highlighted an issue I was wondering about: Whether to carry out the cluster analysis on the original data or on my PCA output. The fact that the derived new variables lack interpretability certainly gives me pause to reconsider my approach. Thank you for clarifying.

If I were to carry out the cluster analysis on the derived variables, is it possible to extract/identify the original variables post-clustering, or are they lost? eg. If I were to carry out the cluster analysis on the Principal Components, the clusters themselves would no longer have any real meaning regarding the other organisms I referred to (ie. the ones that displayed certain physical characteristics from the original dataset, but for which the measurements didn't exist). The reason being, that the clusters would be made up of derived variables, not real-world physical descriptor variables. Is that correct?

@ Paul Siegel Thanks for the words of warning. My data are not categorical, but I take your point. I will look at the other approaches you suggested.

@Frank Harrell I don't use R, only matlab and would like to keep my code to just one language...I will certainly look at the function code/reading you suggest though.

@DJohnson. Thanks! I'll give your methodology a go.

Answer 1

Also consider sparse principal component analysis, and redundancy analysis. The latter is implemented in the R Hmisc package redun function and involves attempting to predict each predictor from all the other predictors. It handles the "wings" issue discussed above.

Answer 2

I'll first remark that conventional PCA is not so well adapted to categorical features (such as whether or not an organism has wings). The reason is that the principal components are generally nontrivial linear combinations of the input features, and it's not always clear what that should mean. For instance the first principal component could be composed of something like $45\%$ "has wings", $30\%$ "feathered", and $25\%$ "6 legs" (these percentages can be computed from the first eigenvector). It would be difficult to use this to eliminate a feature because secretly the "right" features are "has wings and feathers" (birds) and "has wings and six legs" (insects). With enough data some sort of PCA can surface this insight, but it's not always obvious in practice.

That said, there is some work in this direction; a good buzzword is Multiple correspondence analysis. The more general problem that you're considering is feature selection, and there are many good approaches for eliminating redundancy in features subject to certain constraints; given that you're not working with an astronomical number of observations, mRMR might be good for your problem.

Answer 3

Having done the two step exercise of PCA followed by clustering more than a few times, I have developed a strong POV. First, there are lots of good reasons for smoothing your inputs with PCA -- most importantly, redundancy is removed. Next and as @hssay notes, the resulting PCA is a linear combination of all of the inputs. Identifying a subset of features that load maximally on them and retaining only that subset for the cluster solution would destroy variance. Given that, my recommendation is that you use the complete set of components as input to your clustering algorithm.

Then there's your question of interpretation. It is a fact in the applied world that people (teams) can, will and do spend enormous amounts of time on interpreting the components. To me, this is a waste of time since they are merely a means to an end...the end being a "good" cluster solution.

Once you have generated a partitioning of your information that has good statistical properties, create an output file containing the cluster assignments for each object in your data. Next, generate a back-end interpretation based on the original features that ignores the components (since you don't care what they mean). This back-end analysis can consist of a spreadsheet based on the new output file that has columns for the clusters and the rows are an appropriate measure of central tendency, e.g., mean, median, mode, whatever.

To facilitate the interpretation, add an index column for each cluster that represents the ratio of the cluster value to the overall (or grand) value -- include the values for the overall data in your sheet in a separate column. By multiplying that ratio by 100 (and rounding), you create a new heuristic that is kind of like an informal t-test or an IQ score. Indexes between 80 and 120 would be considered "normal" behavior, 120+ is a feature or behavior that is distinctively true for a cluster while indexes 80 and less are features or behaviors that are not representative of that cluster. The more extreme the index, the more that cluster deviates from normative behavior. Just use caution interpreting small values in the denominator as the indexes can get quite large. Another problem with this "quick and dirty" approach to interpretation occurs when some of your values are negative. Negative indexes need more careful consideration.

The fact is that people typically don't care and don't want to know how you got to a solution. They just want the answer. Sometimes they're willing to work with you on getting to a final answer, other times they just leave the whole thing up to you.

Of course, this discussion ignores the question of what a "good" cluster solution is. That's another story.

Answer 4

Specifically on the question of interpreting the result: you cannot generally pinpoint to individual variables contributing maximum variance. What the result of 3 components contributing maximum variance is this: there are 3 new variables derived by taking a linear combination of original variables which is found to represent maximum variance. This is a known problem with PCA: you loose interpretation power. The so called factors derived by combining different variables may not have any intuitive meaning.

One recommendation now is to build your subsequent model (clustering) on the derived variables data. So rather than building clusters on 20 column data, you'll build it on 3 column data.