I'm performing analysis for a colleague who has data from two biological conditions. Since clustering is a standard in our field, she'd like to perform clustering on each condition separately, with the goal of identifying datapoints that change clusters between conditions. For example:

enter image description here

Here, we'd want to flag datapoint 3 as changing clusters. This seems straightforward (look how simple the example is!).

However, there's extra complexity because I'm required to choose the number of clusters a priori, and this changes the result. For example if there's one cluster, datapoint 3 would not be flagged in the example above; if there are three clusters, other datapoints would also be flagged. So, is clustering a viable/appropriate method for doing what we want? If so, is there a conventional method that doesn't require choosing the number of clusters beforehand?

  • 2
    $\begingroup$ To detect changes you don't need clustering at all. Why the idea of clusters haunts you? $\endgroup$
    – ttnphns
    Jan 18 '17 at 8:59
  • 1
    $\begingroup$ I, like @ttnphns think clustering doesn't look like it buys you anything here. Could probably make it work, but there are probably better methods. Is there jitter in the data, i.e. do each of the points move a little from case A to case B? $\endgroup$
    – Matt L.
    Jan 18 '17 at 14:14
  • $\begingroup$ @ttnphns I'm curious if clustering could be used here, and, if it could, a good way to use it. I don't mean to argue it's the best method. $\endgroup$ Jan 18 '17 at 19:04
  • $\begingroup$ @MattL. Yes, there is jitter between A and B (experimental noise and effects from the conditions). $\endgroup$ Jan 18 '17 at 19:05
  • $\begingroup$ When moving from condition A to condition B, one source of variability is change of 'underlying cluster'. You also mention jitter/experimental noise. Is this something like additive noise on top of a true/underlying position, or like drawing an independent sample from the same underlying cluster, or something else? It seems like the fundamental problem is how to distinguish between these sources of variability, so knowing more about them would help. Can you tell us anything more about the data or its properties? $\endgroup$
    – user20160
    Jan 19 '17 at 5:32

I thought up one possible solution utilizing clustering.

  1. Cluster case A
  2. For case B use same cluster assignments.
  3. Calculate the distance to the cluster center for each point in A.
  4. Calculate distance from cluster points to cluster centers in B.
  5. Compare A to B and set some threshold where if a distance exceeds X a flag is set. This would be your data point(s) of interest.

This doesn't really tell you if a cluster switched, but it does tell you if a point(s) have exceed some threshold from their assigned cluster. Might be interesting to include inter-cluster distances too.


If you want to perform clustering without setting number of clusters, then non-parametric methods are something you may try. In particular, you can consider using dirichlet process.

Google give me the following material, https://www.cse.buffalo.edu/~jcorso/t/2009S_555/files/lecture7.dirichlet.pdf.

  • 2
    $\begingroup$ I don't think the OP is asking about determining the number of clusters. It is a interesting link though. $\endgroup$ Jan 18 '17 at 1:19
  • $\begingroup$ @altria Thank you. Yes, Michael Chernick is correct. I'm more asking whether clustering is the right tool for this analysis. $\endgroup$ Jan 18 '17 at 5:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.