Choosing the number of clusters - clustering validation criterions vs domain theoretical considerations

Question

I often face the issue of having to choose a k number of clusters. The partition I end up choosing is more often based on visual and theoretical concerns rather than quality criteria.

I have two main questions.

The first concerns the general idea of clusters quality. From what I understand criteria, such as the "elbow", are suggesting an optimal value in reference to a cost function. The issue I have with this framework is that the optimal criteria is blind to theoretical consideration, so that there are some degree of complexity (related to your field of study) that would always want in your final groups/clusters.

Moreover, as explained here the optimal value is also related to "downstream purpose" constraints (such as economic constraints), so consideration of what you are going to do with the clustering matters.

One constraint obviously that one faces is to find meaningful/interpretable clusters, and the more clusters you have the more difficult it is to interpret them.

But this is not always the case, very often I find that 8, 10 or 12 clusters are the minimum "interesting" number of clusters I would like to have in my analysis.

However, very often criteria such as the elbow suggest much fewer clusters, generally 2,3 or 4.

Q1. What I would like to know is what is the best line of argument when you decide to choose more clusters rather than the solution proposed by a certain criteria (such as the elbow). Intuitively, the more should always be better when there are no constraints (such as the intelligibility of the groups you get or in the coursera example when you have a very large sum of money). How would you argue this in a scientific journal article?

Another way to put this, is to say that once you identified the minimum number of clusters (with these criteria), should you even have to justify why you picked more clusters than that? Shouldn't justification come only when choosing the minimal meaningful amount of clusters?

Q2. Relatedly, I do not understand how certain quality measures, such as the silhouette, can actually decrease as the number of clusters increase. I don't see in the silhouette a penalisation for the number of clusters, so how can this be? Theoretically, the more clusters you have, the greater is the cluster quality?

# R code 

library(factoextra)

data("iris")
ir = iris[,-5]

# Hierarchical Clustering, Ward.D
# 5 clusters
ec5  = eclust(ir, FUNcluster = 'hclust', hc_metric = 'euclidean', 
              hc_method = 'ward.D', graph = T, k = 5)
# 20 clusters
ec20 = eclust(ir, FUNcluster = 'hclust', hc_metric = 'euclidean', 
              hc_method = 'ward.D', graph = T, k = 20)

a = fviz_silhouette(ec5)  # silhouette plot
b = fviz_silhouette(ec20) # silhouette plot

c = fviz_cluster(ec5)  # scatter plot
d = fviz_cluster(ec20) # scatter plot

grid.arrange(a,b,c,d)

Answer 1

The keys are finding meaningful clusters and what you value in the resulting clusters.

Let me illustrate with a simple example. The example is two Gaussian clusters that are pretty well separated. Using k-means to divide the data into either 2 or 3 clusters we get these partitions:

set.seed(1066)
x = c(rnorm(200,0,1), rnorm(200,6,1))
y = rnorm(400,0,1)
XY = data.frame(x,y)

KM2 = kmeans(XY, 2)
KM3 = kmeans(XY, 3)

par(mfrow=c(1,2))
plot(XY, pch=20, col=KM2$cluster+1, asp=1)
plot(XY, pch=20, col=KM3$cluster+1, asp=1)

Silhouette says that you are better off with two clusters rather than three.

library(cluster)
plot(silhouette(KM2$cluster, dist(XY)))
plot(silhouette(KM3$cluster, dist(XY)))

It is useful to look at why the silhouette went down. First of all, it is easy to see that for the cluster on the right, the silhouette barely changed. The reason for the big drop in average silhouette is the cluster on the left that has been split in two. Why didn't silhouette like that? As I said, you need to look at what the metric favors. For each point, silhouette compares the average distance between the point and the other points in the same cluster with the average distance between that point and the nearest other cluster. When there were two clusters, points in each of the two clusters were well separated from the other cluster. Not so with three clusters. The points in the two clusters on the left are right up against each other. That is how the metric can go down. Silhouette not only rewards clusters where the points in a cluster are close together; it also punishes clusters that are not well separated from each other.

So that gets to the "downstream purpose". There are times when having well separated clusters is not so important. For example, you can use k-means clustering on the colors in an image to group similar colors for image compression. In that case, as long as each cluster is reasonably consistent (compact) it does not matter if sometimes two clusters might be close to each other. However, often people use clustering as a way of understanding more fundamental structure in their data. For example, in the two Gaussians example above, two clusters shows the underlying structure better than three clusters. If you are looking for structure, you want the number of clusters that most closely represents natural groupings in your data. But these are two different goals:

1. a grouping of points where points in the same cluster are near each other and

2. a grouping that also separates different clusters

Your argument that more clusters should always be better is OK as long as you only want points in the same cluster to be close. But that is not good if you are trying to discover underlying structure. The structure is what is in the data. Taking one cluster and calling it two is not an improvement.

Answer 2

Note that, cross validation can be also used in clustering problem.

For example, in K means, increasing number of cluster will always decrease the objective we are fitting. An extreme case would be number of clusters equal to number of data points, and the objective is $0$. But that is an overfitted model and will fail on the testing set.

My suggestion is checking the "clustering quality measure" on hold out testing data set.