Choosing the number of clusters - clustering validation criterions vs domain theoretical considerations

I often face the issue of having to choose a k number of clusters. The partition I end up choosing is more often based on visual and theoretical concerns rather than quality criteria.

I have two main questions.

The first concerns the general idea of clusters quality. From what I understand criteria, such as the "elbow", are suggesting an optimal value in reference to a cost function. The issue I have with this framework is that the optimal criteria is blind to theoretical consideration, so that there are some degree of complexity (related to your field of study) that would always want in your final groups/clusters.

Moreover, as explained here the optimal value is also related to "downstream purpose" constraints (such as economic constraints), so consideration of what you are going to do with the clustering matters.

One constraint obviously that one faces is to find meaningful/interpretable clusters, and the more clusters you have the more difficult it is to interpret them.

But this is not always the case, very often I find that 8, 10 or 12 clusters are the minimum "interesting" number of clusters I would like to have in my analysis.

However, very often criteria such as the elbow suggest much fewer clusters, generally 2,3 or 4.

Q1. What I would like to know is what is the best line of argument when you decide to choose more clusters rather than the solution proposed by a certain criteria (such as the elbow). Intuitively, the more should always be better when there are no constraints (such as the intelligibility of the groups you get or in the coursera example when you have a very large sum of money). How would you argue this in a scientific journal article?

Another way to put this, is to say that once you identified the minimum number of clusters (with these criteria), should you even have to justify why you picked more clusters than that? Shouldn't justification come only when choosing the minimal meaningful amount of clusters?

Q2. Relatedly, I do not understand how certain quality measures, such as the silhouette, can actually decrease as the number of clusters increase. I don't see in the silhouette a penalisation for the number of clusters, so how can this be? Theoretically, the more clusters you have, the greater is the cluster quality?

# R code

library(factoextra)

data("iris")
ir = iris[,-5]

# Hierarchical Clustering, Ward.D
# 5 clusters
ec5  = eclust(ir, FUNcluster = 'hclust', hc_metric = 'euclidean',
hc_method = 'ward.D', graph = T, k = 5)
# 20 clusters
ec20 = eclust(ir, FUNcluster = 'hclust', hc_metric = 'euclidean',
hc_method = 'ward.D', graph = T, k = 20)

a = fviz_silhouette(ec5)  # silhouette plot
b = fviz_silhouette(ec20) # silhouette plot

c = fviz_cluster(ec5)  # scatter plot
d = fviz_cluster(ec20) # scatter plot

grid.arrange(a,b,c,d)


• Methods like WCSS in K means algorithm can be used for taking an optimal value of K for clustering. – avi sharma Apr 7 '18 at 13:51
• Some facets of cluster quality stats.stackexchange.com/a/195481/3277. Also, if you want some general info about internal clustering criterions, please see "Clustering criterions" doc on my web page. – ttnphns Apr 7 '18 at 14:44
• Theoretically, the more clusters you have, the greater is the cluster quality Absolutely no, not necessarily. Most internal clustering criterions (including) Silhouette index, are this or that way "normalized" or calibrated in their formula in the aim to try be extreme at the best number(s) of clusters k, so that k less or greater than that number will yield lower criterion value. "Elbow SSw" criterion is not normalized anyhow, and it is bad one, not worth consideration; use Clinski-Harabasz or Davies-Bouldin its normalizations instead. – ttnphns Apr 7 '18 at 15:06
• what is the best line of argument when you decide to choose more clusters rather than the solution proposed by a certain criteria If you read my facets under the link above you'll understand that there can be no single best nor synthesized arguments. After all, the best argument (for a smaller or bigger k) is its persuasiveness to yourself or the audience. Human decision is not based on arguments, it is arbitrary; argumenting is explaining, to excuse what can never be excused. – ttnphns Apr 7 '18 at 15:14
• WCSS will always decrease as K increases, whether more clusters are appropriate or not. – gung Apr 7 '18 at 17:06

The keys are finding meaningful clusters and what you value in the resulting clusters.

Let me illustrate with a simple example. The example is two Gaussian clusters that are pretty well separated. Using k-means to divide the data into either 2 or 3 clusters we get these partitions:

set.seed(1066)
x = c(rnorm(200,0,1), rnorm(200,6,1))
y = rnorm(400,0,1)
XY = data.frame(x,y)

KM2 = kmeans(XY, 2)
KM3 = kmeans(XY, 3)

par(mfrow=c(1,2))
plot(XY, pch=20, col=KM2$cluster+1, asp=1) plot(XY, pch=20, col=KM3$cluster+1, asp=1)


Silhouette says that you are better off with two clusters rather than three.

library(cluster)
plot(silhouette(KM2$cluster, dist(XY))) plot(silhouette(KM3$cluster, dist(XY)))


It is useful to look at why the silhouette went down. First of all, it is easy to see that for the cluster on the right, the silhouette barely changed. The reason for the big drop in average silhouette is the cluster on the left that has been split in two. Why didn't silhouette like that? As I said, you need to look at what the metric favors. For each point, silhouette compares the average distance between the point and the other points in the same cluster with the average distance between that point and the nearest other cluster. When there were two clusters, points in each of the two clusters were well separated from the other cluster. Not so with three clusters. The points in the two clusters on the left are right up against each other. That is how the metric can go down. Silhouette not only rewards clusters where the points in a cluster are close together; it also punishes clusters that are not well separated from each other.

So that gets to the "downstream purpose". There are times when having well separated clusters is not so important. For example, you can use k-means clustering on the colors in an image to group similar colors for image compression. In that case, as long as each cluster is reasonably consistent (compact) it does not matter if sometimes two clusters might be close to each other. However, often people use clustering as a way of understanding more fundamental structure in their data. For example, in the two Gaussians example above, two clusters shows the underlying structure better than three clusters. If you are looking for structure, you want the number of clusters that most closely represents natural groupings in your data. But these are two different goals:

1. a grouping of points where points in the same cluster are near each other and

2. a grouping that also separates different clusters

Your argument that more clusters should always be better is OK as long as you only want points in the same cluster to be close. But that is not good if you are trying to discover underlying structure. The structure is what is in the data. Taking one cluster and calling it two is not an improvement.

• Thank you for your answer, very informative. Just to come back quickly at my point, let us imagine in your example that the 0 as a particular meaning, sort of a qualitative threshold, and that a partition that would appear there would reveal something meaningful (theoretically). My issue with the silhouette is that in fact, even though the points appear close to each others, they are actually theoretically very far (because 0 is some sort of significant threshold.) In my experience, quite often these qualitative differences in the interpretation of the clusters matter. – giacomo Apr 7 '18 at 9:04
• I do not think that we are disagreeing. I made the example as simply two Gaussian clusters, so the difference between -0.1 and 0.1 is small and the three cluster version is bogus. However, I fully accept that there might be applications in which "x<0:" and "x>0" mean something entirely different and this separation would be meaningful. But you cannot expect a metric like silhouette to know your problem. It only sees the data. So it is your responsibility to interpret the data and the metric in terms of your problem. If the metric does not reflect your problem, it will not be useful. – G5W Apr 7 '18 at 10:47
• Yes I agree. Just another thought, my impression is that what you describe is very much the divisive paradigm, but in the agglomerative paradigm, each individual is first and foremost is one cluster. So, I feel that in this paradigm "the more the better". We are not forcing 2 groups to separate, but 2 individuals to merge in a group. I wonder then how appropriate is the silhouette for agglomerative clustering. What do you think? – giacomo Apr 7 '18 at 10:56
• As mentioned in the answer of @hxd1011, the extreme case is allowing every point to be a cluster. The whole point of clustering is to find some structure beyond that, so presumably "more is better" cannot be completely true; only true up to a point. The goal is to find just the right level of combining points so that you capture the structure without merging distinct groups. At every step, you must ask the question - should I stop here or continue to combine clusters? – G5W Apr 7 '18 at 11:35
• In general, if your clusters are not well separated, silhouette will say to continue combining them. However, silhouette is not even defined for clusters with one point. It won't be applicable at the beginning of an agglomerative process. – G5W Apr 7 '18 at 11:35

Note that, cross validation can be also used in clustering problem.

For example, in K means, increasing number of cluster will always decrease the objective we are fitting. An extreme case would be number of clusters equal to number of data points, and the objective is $0$. But that is an overfitted model and will fail on the testing set.

My suggestion is checking the "clustering quality measure" on hold out testing data set.

• Can you give me a reference for the cross-validation in cluster analysis? – giacomo Apr 7 '18 at 10:56