19
$\begingroup$

Is there a way to determine which features / variables of the dataset are the most important / dominant within a k-means cluster solution?

$\endgroup$
  • 1
    $\begingroup$ How do you define "important/dominant"? Do you mean the most useful to discriminate between clusters? $\endgroup$ – Franck Dernoncourt Nov 26 '13 at 1:30
  • 3
    $\begingroup$ Yes the most useful is what I meant. I think part of my problem with figuring this out is how to word it. $\endgroup$ – user1624577 Nov 26 '13 at 1:39
  • $\begingroup$ Thanks for the clarification. One usual term to designate this issue in machine learning is feature selection. $\endgroup$ – Franck Dernoncourt Nov 26 '13 at 2:01
8
$\begingroup$

One way to quantify the usefulness of each feature (= variable = dimension), from the book Burns, Robert P., and Richard Burns. Business research methods and statistics using SPSS. Sage, 2008. (mirror), usefulness being defined by the features' discriminative power to tell clusters apart.

We usually examine the means for each cluster on each dimension using ANOVA to assess how distinct our clusters are. Ideally, we would obtain significantly different means for most, if not all dimensions, used in the analysis. The magnitude of the F values performed on each dimension is an indication of how well the respective dimension discriminates between clusters.

Another way would be to remove a specific feature and see how this impact internal quality indices. Unlike the first solution, you would have to redo the clustering for each feature (or set of features) you want to analyze.

FYI:

$\endgroup$
  • 4
    $\begingroup$ It is very important to add that in this context one should not take those F (or p) values as indicators of statistical significance (i.e. relative the population), but rather simply as indicators of magnitude of the differences. $\endgroup$ – ttnphns Nov 26 '13 at 7:22
3
$\begingroup$

I can think of two other possibilities that focus more on which variables are important to which clusters.

  1. Multi-class classification. Consider the objects that belong to cluster x members of the same class (e.g., class 1) and the objects that belong to other clusters members of a second class (e.g., class 2). Train a classifier to predict class membership (e.g., class 1 vs. class 2). The classifier's variable coefficients can serve to estimate the importance of each variable in clustering objects to cluster x. Repeat this approach for all other clusters.

  2. Intra-cluster variable similarity. For every variable, calculate the average similarity of each object to its centroid. A variable that has high similarity between a centroid and its objects is likely more important to the clustering process than a variable that has low similarity. Of course, similarity magnitude is relative, but now variables can be ranked by the degree to which they help to cluster the objects in each cluster.

$\endgroup$
0
$\begingroup$

Here is a very simple method. Note that the Euclidean distance between two cluster centers is a sum of square difference between individual features. We can then just use the square difference as the weight for each feature.

Euclidean Distance

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.