1
$\begingroup$

I am working to publish some hierarchical clustering indicating phenotypes in a disease population. I log transformed and min-max normalised my input vairables, and used Ward to identify clusters.

In order to select k (where to cut the tree), I used 3 statistical methods - GAP, elbow and silhouette, and selected the "majority vote" - in this case 5.

Following this analysis (and halfway through writing the publicaiton) I came across the hopkins statistic for which my data gave a very low result - 0.2 - indicating non-clusterable data.

I am slightly confused given the results of the 3 previously used clustering stats, and not sure if I should report this value or not..

Any feedback is welcome!

Improve this question
$\endgroup$
5
  • $\begingroup$ SSw "elbow" criterion is a raw quantity which enters formulae of a number of more sophisticated criterions, such as Gap, Calinski-Harabasz, Davis-Bouldin, Cubic clustering criterion, etc. So, I'd recommend using them / one of them instead. You did use Gap. Silhouette index is a bit different idea and formula than "SSw". I don't know Hopkins index. Please note important that (i) different criterions have different "biases", preferences, (ii) some are higher the better, some are lower the better form. $\endgroup$ – ttnphns Nov 22 '17 at 16:16
  • $\begingroup$ You might want to read about clustering criterions on my web-page (download the named so file there). $\endgroup$ – ttnphns Nov 22 '17 at 16:16
  • $\begingroup$ Ok thanks for the response. I understand that the three methods I used were slightly different, that was generally the idea to try and make the selection more robust. With regards to hopkins, it is a test of "randomness" with 0.5 indicating random distribution, 1 clustered and 0 uniform. I seem to be tending towards a uniformly distrubuted population but perhaps thats due to the similarity of the patients $\endgroup$ – JB1 Nov 22 '17 at 16:27
  • $\begingroup$ I will check your web page :) $\endgroup$ – JB1 Nov 22 '17 at 16:31
  • $\begingroup$ How do you get Hopkin statistic? A value close to zero suggests that the data set is clusterable if using function get_clust_tendency from R package factoextra. $\endgroup$ – yaya Dec 13 '18 at 16:09
1
$\begingroup$

The statistics - all of them - are just heuristics.

They don't guarantee to choose the "correct" k, if anything like that exists at all. So any of them, including Hopkins, may be wrong. They also tend to be sensitive to scaling, so maybe you just scaled the data badly. Hopkins is a test for uniform distribution. If you get a low score, your data looks rather uniform, and probably needs better preprocessing.

Add a column with random uniforms in $[0,10^{100}]$, and all the measures should tell you there are no clusters there.

Short story: clustering cannot be automated. It's an explorative technique, that requires an experienced human to fine tune the data pipeline and interpret the results. Use "optimum k" only as a tool to select candidates to explore first.

Improve this answer
$\endgroup$
3
  • $\begingroup$ "that requires an experienced human" - somewhat experienced, hoping to gain more in this process :) $\endgroup$ – JB1 Nov 27 '17 at 10:42
  • 1
    $\begingroup$ Look at the data. Study some samples. Visualize the data. Try to use the clustering. Don't just rely on these heuristics, they may be wrong. You cannot condense all of this into a single floating point number. $\endgroup$ – Has QUIT--Anony-Mousse Nov 27 '17 at 15:10
  • $\begingroup$ I have done all of that :) visually (heatmap) there are some nice clusters with biological relevance (somewhat). I am just always skeptical of everything until I am 100% sure it makes sense! Thanks for your help. I will continue to use multiple heuristics. $\endgroup$ – JB1 Nov 27 '17 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.