# Ever increasing silhouette width and Mantel statistics when seeking optimal number of clusters in hierarchical agglomerative clustering

When wanting to identify the optimum number of clusters in my hierarchical agglomerative clustering attempt (UPGMA and complete linkage), I obtain ever increasing average silhouette widths (Rousseeuw quality index) and Mantel correlation coefficients with increasing number of clusters. Hoping to obtain a limited amount of well interpretable clusters, this is clearly not what I want. Have I gone wrong or is this a true result? And if so: Should I use another tool to identify interpretable clusters or choose an arbitrary k?

Aim of the project is to identify clusters of 10,000 descriptors in a data set with 750 objects. I follow the procedure outlined in Numerical Ecology with R by Borcard, Gillet and Legendre (2011). I have tested clustering based on absolute (count) and binary (presence/absence) versions of the data set; both lead to the described issue. UPGMA and complete linkage clusters were identified as best representations of the data using cophenetic correlation and Gower distance.

When I then run computation and plotting of silhouette widths, average silhouette width ever increases with the number of groups (at least within the range tested - I stopped the calculations at 600+ groups). The same accounts for the comparison between the Jaccard distance matrix and the binary matrices representing partitions of the hierarchical clusters: Their correlation is constantly rising with k, the number of groups.

2) Have a look at how many elements are in each of your clusters. In my case, with the Gap statistic giving monotonically better fits with increasing k, the increasing number of clusters was just overfitting -- each new cluster was peeling off a single observation into its own cluster. The consensus matrices from consensus clustering (library ConsensusClusterPlus in R) can also offer a clue. Does the heatmap look block diagonal for any k, or does increasing k just give tiny blocks along the diagonal with the rest of the data in a large fuzzy square?