Why clustering metrics are worse while adding some features?

Question

I am facing something unexpected at first sight and would like to know if you could share some insights.

Basically, I have performed a clustering on both qualitative and quantitative data using the gower distance as dissimilarity metric (with daisy()), which I fed into the agglomerative clustering function hclust(), setting the complete linkage option. To evaluate the clustering performance, I used the function cluster.stats() which raises many metrics like the squared sum of distance within cluster, average silhouette width,... But I used mainly the within sum of squares.

My issue is the following: while adding some features to my input table for clustering, I expected the WSS to be lower (or at least around the same values) because I have more chance for a better discrimination of individuals. It turns out that the other way around: the WSS is lots higher.

My opinion: the more features into it, the more chance the individuals to be considered as different and the bigger the distances. As a result, I also need more clusters to reach the same performance level. What do you think? If my opinion is true, then I would need to calculate like a feature importance score to get rid of useless features. Any recommendation on how to do it?

Other information: quantitative data were scaledfor equal variance.

I cannot provide you with the data as its private.

Answer 1

You cannot compare SSQ value or distances with different feature sets (or data, for that matter).

It's trivial to see when you consider the definition. It's a sum over different columns... If you add more columns, the sum cannot decrease.

This holds for almost any such measure, that comparing different feature sets or otherwise different data does not work. They are just heuristics, not objective measures of quality.

What you could do is to always compute the measures on the entire data (all features). Then an algorithm given more columns will likely find a "better" clustering than one given fewer columns. But that again is to be expected...