# What are my options for choosing the number of appropriate clusters when gap statistic is monotonically increasing?

I have performed a clustering analysis and I am attempting to choose the optimal number of clusters for my data. I've calculated the gap statistic (using clusGap() in R) and the statistic does not have a local maximum over the # of clusters I examined (k = 1-20). See the plot of the gap statistic below. For my purposes, it is intractable to have a large number of clusters. (I am analyzing hillslope cross-sectional elevation data to identify geomorphic archetypes of hillslopes, but it wouldn't really be useful to have 20+ archetypes to discuss.) So, given that my data do not have a readily apparent "optimum" number of clusters, what are some possible approaches to choosing the number of clusters my analysis will use in the end? Is it appropriate to pick the value that occurs at the plateau around k=9? Is this indicative of there being excessive within-cluster variability, perhaps making this approach not very useful?

EDIT: To clarify, the data that I am clustering is a pairwise distance matrix of hillslope shapes, using dynamic time warping distance as the measure. ## 1 Answer

The gap statistic rule says you should pick k=4 here. (It's not just about picking the maximum, the error bars serve a purpose!)

Although based on the plot, maybe you need to further improve your data preprocessing. K-means is very sensitive to scale, and since your data is likely having variables of mixed scale, I am not convinced k-means is a good choice (not that the gap statistic uses a good null model for your kind of data). You should at least try alternatives such as GMM, and then perform some domain specific test which of these results is most useful to you. Because there is not "the optimal k".

• Thanks for your comments. To clarify, as I did in my edit above, the data that are being clustered are pairwise distances between hillslope shapes, as measured by dynamic time warping. So there isn't a scale mismatch, and it seems to me that the null hypothesis of the gap statistic is reasonable on its face. But I am open to arguments otherwise! – user278411 Nov 25 '19 at 14:39
• The argument for k=4 is: the smallest k included in the next k's error bar (no substantial increase then anymore). – Has QUIT--Anony-Mousse Nov 26 '19 at 23:16