Ever increasing silhouette width and Mantel statistics when seeking optimal number of clusters in hierarchical agglomerative clustering

Question

When wanting to identify the optimum number of clusters in my hierarchical agglomerative clustering attempt (UPGMA and complete linkage), I obtain ever increasing average silhouette widths (Rousseeuw quality index) and Mantel correlation coefficients with increasing number of clusters. Hoping to obtain a limited amount of well interpretable clusters, this is clearly not what I want. Have I gone wrong or is this a true result? And if so: Should I use another tool to identify interpretable clusters or choose an arbitrary k?

Aim of the project is to identify clusters of 10,000 descriptors in a data set with 750 objects. I follow the procedure outlined in Numerical Ecology with R by Borcard, Gillet and Legendre (2011). I have tested clustering based on absolute (count) and binary (presence/absence) versions of the data set; both lead to the described issue. UPGMA and complete linkage clusters were identified as best representations of the data using cophenetic correlation and Gower distance.

When I then run computation and plotting of silhouette widths, average silhouette width ever increases with the number of groups (at least within the range tested - I stopped the calculations at 600+ groups). The same accounts for the comparison between the Jaccard distance matrix and the binary matrices representing partitions of the hierarchical clusters: Their correlation is constantly rising with k, the number of groups.

Answer 1

Late, and partial. Take this as a suggestion, not an answer, as I'm also just learning this field right now.

1) Evaluate your number of clusters with a measure such as the Gap statistic (Estimating the number of clusters in a data set via the gap statistic, Tibshirani, R., Walther, G., Hastie, T., 2001. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63, 411–423. https://doi.org/10.1111/1467-9868.00293 https://web.stanford.edu/~hastie/Papers/gap.pdf), prediction strength (Cluster Validation by Prediction Strength, Tibshirani, R., Walther, G., 2005. Journal of Computational and Graphical Statistics 14, 511–528. https://doi.org/10.1198/106186005X59243, http://statweb.stanford.edu/~gwalther/predictionstrength.pdf), or several others, which includes an evaluation of k = 1 -- i.e., no good clustering -- as an option.

2) Have a look at how many elements are in each of your clusters. In my case, with the Gap statistic giving monotonically better fits with increasing k, the increasing number of clusters was just overfitting -- each new cluster was peeling off a single observation into its own cluster. The consensus matrices from consensus clustering (library ConsensusClusterPlus in R) can also offer a clue. Does the heatmap look block diagonal for any k, or does increasing k just give tiny blocks along the diagonal with the rest of the data in a large fuzzy square?

3) Also -- if the y axis scale in the first plot is to be believed, those silhouette widths are all tiny?

(Finally, with 10,000 predictors, are you using Euclidean distance? On the Surprising Behavior of Distance Metrics in High Dimensional Space, Aggarwal, C.C., Hinneburg, A., Keim, D.A., 2001. in: Database Theory — ICDT 2001, Lecture Notes in Computer Science. Presented at the International Conference on Database Theory, Springer, Berlin, Heidelberg, pp. 420–434. https://doi.org/10.1007/3-540-44503-X_27, https://link.springer.com/content/pdf/10.1007%2F3-540-44503-X_27.pdf.)