I often face the issue of having to choose a k number of clusters. The partition I end up choosing is more often based on visual and theoretical concerns rather than quality criteria.
I have two main questions.
The first concerns the general idea of clusters quality. From what I understand criteria, such as the "elbow", are suggesting an optimal value in reference to a cost function. The issue I have with this framework is that the optimal criteria is blind to theoretical consideration, so that there are some degree of complexity (related to your field of study) that would always want in your final groups/clusters.
Moreover, as explained here the optimal value is also related to "downstream purpose" constraints (such as economic constraints), so consideration of what you are going to do with the clustering matters.
One constraint obviously that one faces is to find meaningful/interpretable clusters, and the more clusters you have the more difficult it is to interpret them.
But this is not always the case, very often I find that 8, 10 or 12 clusters are the minimum "interesting" number of clusters I would like to have in my analysis.
However, very often criteria such as the elbow suggest much fewer clusters, generally 2,3 or 4.
Q1. What I would like to know is what is the best line of argument when you decide to choose more clusters rather than the solution proposed by a certain criteria (such as the elbow). Intuitively, the more should always be better when there are no constraints (such as the intelligibility of the groups you get or in the coursera example when you have a very large sum of money). How would you argue this in a scientific journal article?
Another way to put this, is to say that once you identified the minimum number of clusters (with these criteria), should you even have to justify why you picked more clusters than that? Shouldn't justification come only when choosing the minimal meaningful amount of clusters?
Q2. Relatedly, I do not understand how certain quality measures, such as the silhouette, can actually decrease as the number of clusters increase. I don't see in the silhouette a penalisation for the number of clusters, so how can this be? Theoretically, the more clusters you have, the greater is the cluster quality?
# R code library(factoextra) data("iris") ir = iris[,-5] # Hierarchical Clustering, Ward.D # 5 clusters ec5 = eclust(ir, FUNcluster = 'hclust', hc_metric = 'euclidean', hc_method = 'ward.D', graph = T, k = 5) # 20 clusters ec20 = eclust(ir, FUNcluster = 'hclust', hc_metric = 'euclidean', hc_method = 'ward.D', graph = T, k = 20) a = fviz_silhouette(ec5) # silhouette plot b = fviz_silhouette(ec20) # silhouette plot c = fviz_cluster(ec5) # scatter plot d = fviz_cluster(ec20) # scatter plot grid.arrange(a,b,c,d)