26
$\begingroup$

Is there a way to determine which features / variables of the dataset are the most important / dominant within a k-means cluster solution?

$\endgroup$
4
  • 1
    $\begingroup$ How do you define "important/dominant"? Do you mean the most useful to discriminate between clusters? $\endgroup$ Nov 26, 2013 at 1:30
  • 3
    $\begingroup$ Yes the most useful is what I meant. I think part of my problem with figuring this out is how to word it. $\endgroup$ Nov 26, 2013 at 1:39
  • $\begingroup$ Thanks for the clarification. One usual term to designate this issue in machine learning is feature selection. $\endgroup$ Nov 26, 2013 at 2:01
  • $\begingroup$ One of popular internal clustering criteria, Ratkowski-Lance, can evaluate the "quality" of a cluster partition on the level of each variable separately, thus measuring the contribution or importance of it. It does it on the basis of one-way ANOVA. This is an approach identical or similar to that described below by Frank. $\endgroup$
    – ttnphns
    Aug 22, 2021 at 11:25

4 Answers 4

16
$\begingroup$

One way to quantify the usefulness of each feature (= variable = dimension), from the book Burns, Robert P., and Richard Burns. Business research methods and statistics using SPSS. Sage, 2008. (mirror), usefulness being defined by the features' discriminative power to tell clusters apart.

We usually examine the means for each cluster on each dimension using ANOVA to assess how distinct our clusters are. Ideally, we would obtain significantly different means for most, if not all dimensions, used in the analysis. The magnitude of the F values performed on each dimension is an indication of how well the respective dimension discriminates between clusters.

Another way would be to remove a specific feature and see how this impact internal quality indices. Unlike the first solution, you would have to redo the clustering for each feature (or set of features) you want to analyze.

FYI:

$\endgroup$
3
  • 4
    $\begingroup$ It is very important to add that in this context one should not take those F (or p) values as indicators of statistical significance (i.e. relative the population), but rather simply as indicators of magnitude of the differences. $\endgroup$
    – ttnphns
    Nov 26, 2013 at 7:22
  • 1
    $\begingroup$ wouldnt the ANOVA imply that each cluster is normally distributed wrt each of its features, and also that each cluster has the same variance amongst each feature ? $\endgroup$
    – quant
    Sep 28, 2020 at 14:19
  • 1
    $\begingroup$ Also, have you considered this: stats.stackexchange.com/questions/116294/… ? Unless I misunderstand something $\endgroup$
    – quant
    Sep 28, 2020 at 16:31
8
$\begingroup$

I can think of two other possibilities that focus more on which variables are important to which clusters.

  1. Multi-class classification. Consider the objects that belong to cluster x members of the same class (e.g., class 1) and the objects that belong to other clusters members of a second class (e.g., class 2). Train a classifier to predict class membership (e.g., class 1 vs. class 2). The classifier's variable coefficients can serve to estimate the importance of each variable in clustering objects to cluster x. Repeat this approach for all other clusters.

  2. Intra-cluster variable similarity. For every variable, calculate the average similarity of each object to its centroid. A variable that has high similarity between a centroid and its objects is likely more important to the clustering process than a variable that has low similarity. Of course, similarity magnitude is relative, but now variables can be ranked by the degree to which they help to cluster the objects in each cluster.

$\endgroup$
0
$\begingroup$

I faced this problem before and developed two possible methods to find the most important features responsible for each K-Means cluster sub-optimal solution.

  1. Focusing on each centroid’s position and the dimensions responsible for the highest Within-Cluster Sum of Squares minimization

  2. Converting the problem into classification settings (Inspired by the paper: "A Supervised Methodology to Measure the Variables Contribution to a Clustering").

I have written a detailed article here Interpretable K-Means: Clusters Feature Importances. GitHub link is included as well if you want to try it. Hope this helps!

$\endgroup$
-1
$\begingroup$

Here is a very simple method. Note that the Euclidean distance between two cluster centers is a sum of square difference between individual features. We can then just use the square difference as the weight for each feature.

Euclidean Distance

$\endgroup$
1
  • 1
    $\begingroup$ What is c1 and c2, clusters center? $\endgroup$
    – abdoulsn
    Jul 1, 2020 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.