A recent paper used OPTICS reachability plot prior to clustering to determine the clustering method.

Based on their results they felt the reachability plot advocated for the use of k-means based algorithm over a hierarchical clustering approach.

enter image description here

"The OPTICS plots shows a smooth rise in reachability distance (as opposed to well demarcated sets). This implies that a partioning approach such as consensus K means clustering is the preferred statistical algorithm, as opposed to a clustering approach such as hierarchical clustering."

Questions: Is there any basis for this assertion, either theoretically or based on prior literature? I couldn't find either and the plots seem expected with high-dimensional dataset

Paper Derivation, Validation, and Potential Treatment Implications of Novel Clinical Phenotypes for Sepsis. JAMA May 19. 2019 doi:10.1001/jama.2019.5791 https://jamanetwork.com/journals/jama/fullarticle/2733996

Descriptions of OPTICS "OPTICS is able to detect natural clusters with various densities and is not overly sensitive about use-selected tuning parameters. It generates a reachability plot that can provide an overall visualization of data structure and help guide towards appropriate clustering methods. In general, an OPTICS plot that is smooth is more suitable for partitioning, whereas a plot that is jagged and stepped is more suitable for clustering."

  • 1
    $\begingroup$ Is there a open (or pre-print) version of this? I tried to look it up but I cannot answer the question without looking to the paper. That said, +1 to @Anony-Mousse, the fact that no density-based clusters emerges based on the reachability plot does not suggest that consensus $k$-means is the right answer. For all we know it also suggests that there is no obvious clustering of the data. Or maybe the MinPts was unsuitable. :) $\endgroup$ – usεr11852 May 26 '19 at 23:16
  • $\begingroup$ Thank you for comment! Appreciated. (1) Sorry: still behind a firewall apparently. Added some more detail to question but even the paper is vague (2) The paper doesn't mention MinPts but instead makes the general comment that the OPTICS is not overerly sensitive to tuning parameters $\endgroup$ – charles May 27 '19 at 1:45

I am not aware of such a result, and I don't fully agree with the conclusion. I think it is a good idea to study the OPTICS plot prior to clustering, though.

This plot suggests that there are no density-based clusters in the data, as preprocessed. So it contradicts the hypothesis that there are separate genotypes nthst could be identified with the data. IMHO it does not suggest to use k-means, as you want well separated clusters also in k-means. If you have a data set where k-means reliably produces good results, it will usually also have multiple valleys in the plot.

Now they didn't just use k-means, but a "consensus" variant that in my opinion has very unclear semantics. It involves a hierarchical clustering step, yet they seem to choose k based on this. To me, that is a bit like messing with the code until you got a result that you liked, then trying to argue why you did it this way afterwards.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.