I often face the issue of having to choose a k number of clusters. The partition I end up choosing is more often based on visual and theoretical concerns rather than quality criteria.
I have two main questions.
The first concerns the general idea of clusters quality. From what I understand criteria, such as the "elbow", are suggesting an optimal value in reference to a cost function. The issue I have with this framework is that the optimal criteria is blind to theoretical consideration, so that there are some degree of complexity (related to your field of study) that would always want in your final groups/clusters.
Moreover, as explained here the optimal value is also related to "downstream purpose" constraints (such as economic constraints), so consideration of what you are going to do with the clustering matters.
One constraint obviously that one faces is to find meaningful/interpretable clusters, and the more clusters you have the more difficult it is to interpret them.
But this is not always the case, very often I find that 8, 10 or 12 clusters are the minimum "interesting" number of clusters I would like to have in my analysis.
However, very often criteria such as the elbow suggest much fewer clusters, generally 2,3 or 4.
Q1. What I would like to know is what is the best line of argument when you decide to choose more clusters rather than the solution proposed by a certain criteria (such as the elbow). Intuitively, the more should always be better when there are no constraints (such as the intelligibility of the groups you get or in the coursera example when you have a very large sum of money). How would you argue this in a scientific journal article?
Another way to put this, is to say that once you identified the minimum number of clusters (with these criteria), should you even have to justify why you picked more clusters than that? Shouldn't justification come only when choosing the minimal meaningful amount of clusters?
Q2. Relatedly, I do not understand how certain quality measures, such as the silhouette, can actually decrease as the number of clusters increase. I don't see in the silhouette a penalisation for the number of clusters, so how can this be? Theoretically, the more clusters you have, the greater is the cluster quality?
# R code
library(factoextra)
data("iris")
ir = iris[,-5]
# Hierarchical Clustering, Ward.D
# 5 clusters
ec5 = eclust(ir, FUNcluster = 'hclust', hc_metric = 'euclidean',
hc_method = 'ward.D', graph = T, k = 5)
# 20 clusters
ec20 = eclust(ir, FUNcluster = 'hclust', hc_metric = 'euclidean',
hc_method = 'ward.D', graph = T, k = 20)
a = fviz_silhouette(ec5) # silhouette plot
b = fviz_silhouette(ec20) # silhouette plot
c = fviz_cluster(ec5) # scatter plot
d = fviz_cluster(ec20) # scatter plot
grid.arrange(a,b,c,d)
Theoretically, the more clusters you have, the greater is the cluster quality
Absolutely no, not necessarily. Most internal clustering criterions (including) Silhouette index, are this or that way "normalized" or calibrated in their formula in the aim to try be extreme at the best number(s) of clusters k, so that k less or greater than that number will yield lower criterion value. "Elbow SSw" criterion is not normalized anyhow, and it is bad one, not worth consideration; use Clinski-Harabasz or Davies-Bouldin its normalizations instead. $\endgroup$what is the best line of argument when you decide to choose more clusters rather than the solution proposed by a certain criteria
If you read my facets under the link above you'll understand that there can be no single best nor synthesized arguments. After all, the best argument (for a smaller or bigger k) is its persuasiveness to yourself or the audience. Human decision is not based on arguments, it is arbitrary; argumenting is explaining, to excuse what can never be excused. $\endgroup$