3
$\begingroup$

I am facing something unexpected at first sight and would like to know if you could share some insights.

Basically, I have performed a clustering on both qualitative and quantitative data using the gower distance as dissimilarity metric (with daisy()), which I fed into the agglomerative clustering function hclust(), setting the complete linkage option. To evaluate the clustering performance, I used the function cluster.stats() which raises many metrics like the squared sum of distance within cluster, average silhouette width,... But I used mainly the within sum of squares.

My issue is the following: while adding some features to my input table for clustering, I expected the WSS to be lower (or at least around the same values) because I have more chance for a better discrimination of individuals. It turns out that the other way around: the WSS is lots higher.

My opinion: the more features into it, the more chance the individuals to be considered as different and the bigger the distances. As a result, I also need more clusters to reach the same performance level. What do you think? If my opinion is true, then I would need to calculate like a feature importance score to get rid of useless features. Any recommendation on how to do it?

Other information: quantitative data were scaledfor equal variance.

I cannot provide you with the data as its private.

$\endgroup$
2
  • 1
    $\begingroup$ Welcome to the site. Yes, adding more features can mean adding more noise. For qualitative variables you can use the mutual information to filter variables, for quantitative variables, you can filter by standard deviation. $\endgroup$
    – llrs
    Nov 21, 2018 at 9:43
  • $\begingroup$ Using SS based validation criteria is silly with nominal, qualitative data. Besides, as you add features, you are adding SS by definition. $\endgroup$
    – ttnphns
    Nov 26, 2018 at 22:32

1 Answer 1

1
$\begingroup$

You cannot compare SSQ value or distances with different feature sets (or data, for that matter).

It's trivial to see when you consider the definition. It's a sum over different columns... If you add more columns, the sum cannot decrease.

This holds for almost any such measure, that comparing different feature sets or otherwise different data does not work. They are just heuristics, not objective measures of quality.

What you could do is to always compute the measures on the entire data (all features). Then an algorithm given more columns will likely find a "better" clustering than one given fewer columns. But that again is to be expected...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.