2
$\begingroup$

How can i know for which clustering algorithms (with a parameter that represents number of clusters) it makes sense to use the Gap statistic? I've read in the paper by Tibshirani, Walter & Hastie that

It is designed to be applicable to virtually any clustering method.

But then the authors proceed in the theoretical part to

For simplicity [...] focus on the widely used K-means clustering procedure.

My question is, what are the procedures for which it can really be applied? What changes do i need to make (if any) when applying the Gap statistics to other procedures? Should i choose different measures of distance (as opposed to defaulting to the euclidian used for K-means) for different procedures?

To provide a specific list of algorithms i am curious about:

  • k-modes & k-prototypes - Does it make sense to use Gap statistic with a different distance measure? Specifically, using a distances related to the cost functions used by these two algorithms?
  • Ward hierarchical clustering
  • Spectral clustering - is there any way to make gap statistic useful for selection of clusters in spectral clustering? I am not really sure if i should just swap euclidian distance for some other measure (if so, which?), keep using euclidian distance, or there simply is not a way to make gap statistic meaningful.

I am sure that after reading my question the first thought will be that it really depends on what i mean by the words "useful", "meaningful", "right" and "work", but putting this aside, i am looking for systematic ways how to choose number of clusters. I would like these ways of finding number of clusters not to be irrational and would like to avoid a scenario where i do something that is widely considered a bad approach.

$\endgroup$
3
  • 1
    $\begingroup$ Gap clustering criterion is suitable to validate cluster solutions of any cluster analysis. The index is akin to ANOVA-based ones such as Calinski-Carabasz (stats.stackexchange.com/a/358937/3277). Therefore, it is for a quantitative dataset. $\endgroup$
    – ttnphns
    Aug 23, 2020 at 10:12
  • $\begingroup$ @ttnphns And is it suitable even if i use the euclidian distance as a measure of distance? Or should i use the distance that is used by the respective algorithm? I suppose that at the very least for categorical variables the euclidian distance doesn't make much sense and therefore at least for k-modes and k-prototypes i need a to use different measure of distance? $\endgroup$
    – ira
    Aug 23, 2020 at 11:42
  • $\begingroup$ Gap is OK for euclidean distance (if a program can compute it from the distance matrix at all - I don't remember now). Gap is not for categorical data. $\endgroup$
    – ttnphns
    Aug 23, 2020 at 14:54

1 Answer 1

-1
$\begingroup$

It can be only used for K-means, (Partitioning Around Medoids (PAM), and Clustering Large Applications (CLARA). Reference: https://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/#gap-statistic-method However, we can use Elbow method and Average silhouette width for almost all the clustering algorithms

$\endgroup$
2
  • $\begingroup$ There is no reason why it should not be applicable for Ward's method, which uses the same loss function as k-means. Regarding further methods it can be easily modified to adapt the gap statistic to other distances. In fact this is advisable for PAM if a distance other than Euclidean is used. Whether "gap applies to method X" is a correct statement largely relies on how many adaptation you allow without changing the name to something else than "gap statistic". $\endgroup$ Jan 19 at 13:47
  • $\begingroup$ By the way the linked reference doesn't say that gap cannot be applied to other clustering methods, and the question whether this could make sense is very different from the question whether any specific software can be used to do that. $\endgroup$ Jan 19 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.