When wanting to identify the optimum number of clusters in my hierarchical agglomerative clustering attempt (UPGMA and complete linkage), I obtain ever increasing average silhouette widths (Rousseeuw quality index) and Mantel correlation coefficients with increasing number of clusters. Hoping to obtain a limited amount of well interpretable clusters, this is clearly not what I want. Have I gone wrong or is this a true result? And if so: Should I use another tool to identify interpretable clusters or choose an arbitrary k?
Aim of the project is to identify clusters of 10,000 descriptors in a data set with 750 objects. I follow the procedure outlined in Numerical Ecology with R by Borcard, Gillet and Legendre (2011). I have tested clustering based on absolute (count) and binary (presence/absence) versions of the data set; both lead to the described issue. UPGMA and complete linkage clusters were identified as best representations of the data using cophenetic correlation and Gower distance.
When I then run computation and plotting of silhouette widths, average silhouette width ever increases with the number of groups (at least within the range tested - I stopped the calculations at 600+ groups). The same accounts for the comparison between the Jaccard distance matrix and the binary matrices representing partitions of the hierarchical clusters: Their correlation is constantly rising with k, the number of groups.