# Conflicting clustering statistics when chosing optimal k

I am working to publish some hierarchical clustering indicating phenotypes in a disease population. I log transformed and min-max normalised my input vairables, and used Ward to identify clusters.

In order to select k (where to cut the tree), I used 3 statistical methods - GAP, elbow and silhouette, and selected the "majority vote" - in this case 5.

Following this analysis (and halfway through writing the publicaiton) I came across the hopkins statistic for which my data gave a very low result - 0.2 - indicating non-clusterable data.

I am slightly confused given the results of the 3 previously used clustering stats, and not sure if I should report this value or not..

Any feedback is welcome!

• SSw "elbow" criterion is a raw quantity which enters formulae of a number of more sophisticated criterions, such as Gap, Calinski-Harabasz, Davis-Bouldin, Cubic clustering criterion, etc. So, I'd recommend using them / one of them instead. You did use Gap. Silhouette index is a bit different idea and formula than "SSw". I don't know Hopkins index. Please note important that (i) different criterions have different "biases", preferences, (ii) some are higher the better, some are lower the better form. – ttnphns Nov 22 '17 at 16:16
• You might want to read about clustering criterions on my web-page (download the named so file there). – ttnphns Nov 22 '17 at 16:16
• Ok thanks for the response. I understand that the three methods I used were slightly different, that was generally the idea to try and make the selection more robust. With regards to hopkins, it is a test of "randomness" with 0.5 indicating random distribution, 1 clustered and 0 uniform. I seem to be tending towards a uniformly distrubuted population but perhaps thats due to the similarity of the patients – JB1 Nov 22 '17 at 16:27
• I will check your web page :) – JB1 Nov 22 '17 at 16:31
• How do you get Hopkin statistic? A value close to zero suggests that the data set is clusterable if using function get_clust_tendency from R package factoextra. – yaya Dec 13 '18 at 16:09

Add a column with random uniforms in $[0,10^{100}]$, and all the measures should tell you there are no clusters there.