# Reducing the number of variables by PCA vs. by clustering the features

Assume you have much more variables than samples and you want to reduce the number of variables (because you think some of them are redundant). The natural approach to do this is PCA, but is it possible to apply clustering algorithms where you consider the features as samples and try to regroup them? If so: is there a 'better' approach between pca and clustering of the variables?

## migrated from stackoverflow.comMay 9 '14 at 14:40

This question came from our site for professional and enthusiast programmers.

• Note that even if you manage to cluster the features, you will still need to select some "representatives" from each class (in order to reduce the total number of features), which might also pose a problem. – amoeba says Reinstate Monica Dec 29 '14 at 0:32
• Another suggestion, in addition to approaches, mentioned in the answers, is to use factor analysis, which also results in dimensionality reduction, with a potential added benefit of extracting a set of meaningful latent factors (variables). – Aleksandr Blekh Dec 29 '14 at 1:05

Assume you have much more variables than samples and you want to reduce the number of variables

This is a typical scenario in many text mining and bioinformatics problems. The no. of features ($d$) will be very large compared to the samples ($n$) i.e. $d \gg n$. Typically, in text mining, Latent Semantic Analysis (LSA) is more popular way of reducing dimensionality than PCA. This is because LSA tries capture the co-occurrence of features in a set of samples (whereas PCA tries to capture the correlation of features) which helps capturing latent semantics (context based meaning) better. Hence, my first suggestion: Try LSA rather than PCA.

is it possible to apply clustering algorithms where you consider the features as samples and try to regroup them

This is exactly what is done in co-clustering. Both features and samples will be clustered simultaneously. Second suggestion: Try co-clustering.

Third suggestion: You may try feature selection, which works well in similar situations.

PCA works well in reducing dimensionality when you have many redundant features or features which are highly correlated. Since you have not mentioned regarding the correlation among your features, I could not comment on ruling out PCA altogether.

Fourth suggestion: You may still give PCA a try, if the aforementioned stuff didn't solve your problem!

There are many clustering algorithms and manifold learning techniques. Realistically the sensible thing to do with most datasets is to run PCA on it because it is likely that your features have some linearities and share some signal, so running PCA to remove that sort of feature is probably sensible. After that, you can easily try mutliple techniques to determine what is best for your data. You should also read about the No Free Lunch Theorum which basically states that it is impossible to determine the optimal technique for your situation without actually trying something.

Reducing the number of variables by using PCA and then do a clustering on it is generally a bad idea because the clustering structure is not always on the first axes of the PCA.

Let me give an example. If you take a dataset where the rows represent time, you will have a high correlation between close rows and the PCA will result a first axis with high eigenvalue whereas the structure of the clustering might be on the 2nd and 3rd axis.

It has been proved (e.g., Arabie & Hubert, 1994; Chang, 1983; De Soete & Carroll, 1994) that there are more efficient ways to do what you are trying to do. I advice you to use algorithms like reduced kmeans or factorial kmeans.