I am facing something unexpected at first sight and would like to know if you could share some insights.
Basically, I have performed a clustering on both qualitative and quantitative data using the gower distance as dissimilarity metric (with
daisy()), which I fed into the agglomerative clustering function
hclust(), setting the complete linkage option. To evaluate the clustering performance, I used the function
cluster.stats() which raises many metrics like the squared sum of distance within cluster, average silhouette width,... But I used mainly the within sum of squares.
My issue is the following: while adding some features to my input table for clustering, I expected the WSS to be lower (or at least around the same values) because I have more chance for a better discrimination of individuals. It turns out that the other way around: the WSS is lots higher.
My opinion: the more features into it, the more chance the individuals to be considered as different and the bigger the distances. As a result, I also need more clusters to reach the same performance level. What do you think? If my opinion is true, then I would need to calculate like a feature importance score to get rid of useless features. Any recommendation on how to do it?
Other information: quantitative data were scaledfor equal variance.
I cannot provide you with the data as its private.