Estimating the most important features in a k-means cluster partition

Question

Is there a way to determine which features / variables of the dataset are the most important / dominant within a k-means cluster solution?

Answer 1

One way to quantify the usefulness of each feature (= variable = dimension), from the book Burns, Robert P., and Richard Burns. Business research methods and statistics using SPSS. Sage, 2008. (mirror), usefulness being defined by the features' discriminative power to tell clusters apart.

We usually examine the means for each cluster on each dimension using ANOVA to assess how distinct our clusters are. Ideally, we would obtain significantly different means for most, if not all dimensions, used in the analysis. The magnitude of the F values performed on each dimension is an indication of how well the respective dimension discriminates between clusters.

Another way would be to remove a specific feature and see how this impact internal quality indices. Unlike the first solution, you would have to redo the clustering for each feature (or set of features) you want to analyze.

FYI:

• Can a useless feature negatively impact the clustering?
• Can the choice of the measurement units of the features impact the clustering?
• Why vector normalization can improve the accuracy of clustering and classification?
• What are the most commonly used ways to perform feature selection for k-means clustering?

Answer 2

I can think of two other possibilities that focus more on which variables are important to which clusters.

1. Multi-class classification. Consider the objects that belong to cluster x members of the same class (e.g., class 1) and the objects that belong to other clusters members of a second class (e.g., class 2). Train a classifier to predict class membership (e.g., class 1 vs. class 2). The classifier's variable coefficients can serve to estimate the importance of each variable in clustering objects to cluster x. Repeat this approach for all other clusters.

2. Intra-cluster variable similarity. For every variable, calculate the average similarity of each object to its centroid. A variable that has high similarity between a centroid and its objects is likely more important to the clustering process than a variable that has low similarity. Of course, similarity magnitude is relative, but now variables can be ranked by the degree to which they help to cluster the objects in each cluster.

Answer 3

I faced this problem before and developed two possible methods to find the most important features responsible for each K-Means cluster sub-optimal solution.

1. Focusing on each centroid’s position and the dimensions responsible for the highest Within-Cluster Sum of Squares minimization

2. Converting the problem into classification settings (Inspired by the paper: "A Supervised Methodology to Measure the Variables Contribution to a Clustering").

I have written a detailed article here Interpretable K-Means: Clusters Feature Importances. GitHub link is included as well if you want to try it. Hope this helps!

Answer 4

Here is a very simple method. Note that the Euclidean distance between two cluster centers is a sum of square difference between individual features. We can then just use the square difference as the weight for each feature.