I was just wondering if there is a way to check if the clustering found by a specific method (e.g., KMeans, DBSCAN, Mixture Models, ...) is significant. Something in line with, there is 5% chance that the clustering found by the algorithm isn't random.

I am struggling to find the correct method for this. Bootstrapping or permutation testing is not an option, since there is no true label.

  • 1
    $\begingroup$ This would first require a precise definition of a cluster, which may be quite hard for some problems. $\endgroup$ – bi_scholar Jan 8 at 14:06
  • 1
    $\begingroup$ >Something in line with, there is 5% change that the clustering found by the algorithm isn't random. - this is a common misconception about p-values, they do not guarantee that. I'd recommend you to start with Gaussian Mixture Models and BIC/AIC to address your initial question (even if it is not about pvalues) $\endgroup$ – German Demidov Jan 8 at 14:06
  • $\begingroup$ What do you mean by "there is no true label"? Don't you believe that there is such a thing as a true model (then p-values don't make sense) or are you saying that the problem is that you don't observe the true label? $\endgroup$ – Andreas Dzemski Jan 8 at 14:36
  • $\begingroup$ I have a paper where we suggest a p-value for a clustering (arxiv.org/abs/1801.00332). The method is very model specific though (clustering in panels). $\endgroup$ – Andreas Dzemski Jan 8 at 14:43
  • $\begingroup$ @AndreasDzemski , what I tried to say is that I don't observe the true label. I am not sure if there is a true model. I am trying to express whether the clustering found by the algorithm is not just random noise. $\endgroup$ – Th. Jan 8 at 14:46

There is the variance ratio criterion, closely related to the F-test.

But if you interpret it as f-test you get absurd high p-values because the test data not being independent to how the data was generated.

You could literally optimize the model to maximize the p-value, and the results will be meaningless, pure overfitting to this one number.

If you want to test your model, it will need to be on fresh data.

A different strategy is used by the gap statistic. IIRC they generate random data assuming a uniform distribution. If you do this 100 times and that data always clusters worse, then you can at least argue that your input data is not uniform... (It could be a diagonal line, or a Gaussian though, neither of which are good clusters...).

  • $\begingroup$ +1 for gap statistic -- can't get a much more "null" distribution than uniform. $\endgroup$ – Bey Jan 12 at 4:55

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.