I am trying to implement a functionality which can auto compute the optimal number of clusters for a given data. I tried using Average Silhouette Width, Dunn Index, Davis and Bouldin index etc. But all these indices require that I first cluster data and then compute these. But for clustering the data I'll have to use a clustering algorithm. I don't want to do that first. I was wondering if there are some data properties or some indices - like mentioned above - which can be used for my purpose to estimate the number of clusters in data, but which do not require to use of any clustering algorithm. Is it possible to find optimal number of clusters for a given data without or prior using any clustering algorithm?

  • 4
    $\begingroup$ Good luck, since nobody was able to achieve this before! Moreover I don't think that it is even possible without considering the clustering algorithm. However you didn't ask any question, so please edit it to be a question, e.g. "is it possible..?". $\endgroup$ – Tim Sep 11 '17 at 9:54
  • $\begingroup$ An often un-examined premise with this sort of question is a precise characterization what a 'cluster' is for the context of the problem. Without first specifying this in terms of the statistics you have used to model the problem, you cannot possibly hope but to try over a range of cluster amounts and using the minimum number that models the data well. $${}$$ Also, check the related questions on the sidebar. This sort of question has surely already been asked here many times before. $\endgroup$ – Christian Chapman Sep 11 '17 at 10:49
  • 3
    $\begingroup$ You are not the first one who wanted to invent such "beforehand" estimation of the number of clusters. It is (i) a challenging task, (ii) not much worth the candle. SPSS TwoStep cluster analysis (you might search on this site about it) is one of few clustering procedures which has such "auto" determining the optimal number of clusters option. It actually does it not before the clustering but "a halfway" during the clustering process. The efficiency is - "so so", I would say. $\endgroup$ – ttnphns Sep 11 '17 at 11:39
  • 1
    $\begingroup$ That was about find num of clusters in advance. Now about such estimation independent of clustering algorithm used or planned. It is actually impossible, such a clean "atheoretic" or "universal" measure. See footnote here: stats.stackexchange.com/a/195481/3277. $\endgroup$ – ttnphns Sep 11 '17 at 11:56
  • $\begingroup$ If you have MATLAB, i.e. if you want to load your data in MATLAB first then this link will be helpful in your case : sites.google.com/site/kenichikurihara/academic-software/… And you need to give your data in their format i.e. D x N (D : Dimension and N : Data Points). One just need to visit this site and follow their directions. They have developed a MATLAB program/software for Bayesian K-Means which searches the optimal number of clusters. Once you download their MATLAB solution, so you just need to load your data and the program will give you optimal number of cluster $\endgroup$ – Saeed Ullah Sep 12 '17 at 8:32

So you want to compute the optimal number of clusters, but not the actual clusters? Why?

Most likely, this is not possible (but rather, you would need to enumerate all possible subsets, and that does mean clustering the data set).

But the number of clusters is not very meaningful without any restriction on what a "proper" cluster is. But then, again, you are actually finding clusters.

Internal evaluation metrics - which in my opinion are heavily overrated except on toy problems - themselves can be seen as "clustering algorithms" on their own, because they imply an optimal way of partitioning the data. It is just that we don't know any efficient algorithm to find that optimum, so we use e.g. k-means to "guess" possible solutions, as we don't want to try all possible partitions.

| cite | improve this answer | |
  • 2
    $\begingroup$ I agree with the answer but... I just wonder why you like so much downvoting (and not upvoting). The current question, for example deserves not to be downvoted, it is sufficiently interesting question. Well, you may have reasons behind your voting style, but mind please, each downvote, especially of a question of a newcomer, is a potential hurt. Please pardon me for the notice. $\endgroup$ – ttnphns Sep 11 '17 at 20:16
  • $\begingroup$ FWIW, after your edit I could undo the downvote; although I still do believe this question will not be helpful to many other readers. It's very specific, and self-contradictory (clustering, but without clustering). $\endgroup$ – Has QUIT--Anony-Mousse Sep 12 '17 at 14:28
  • 1
    $\begingroup$ I appreciate your comment here, @ttnphns. I understand the sentiment that the question seems confused, & possibly self-contradictory, Anony-Mousse, but many questions have that character here, not just clustering ones. It is a consequence of the OP having an incomplete understanding of the topics being asked about. TBH, this seems like a very intuitive question to me, & I wouldn't be surprised if many other people have some version of this question. $\endgroup$ – gung - Reinstate Monica Sep 14 '17 at 19:15
  • $\begingroup$ I see it as analogous to questions asking for an assumption-free method to test if a distribution differs from the normal, & not recognizing that every test only assesses a specific kind of deviation. Eg, speaking for myself, I immediately recognize, "Internal evaluation metrics ... can be seen as "clustering algorithms" on their own, because they imply an optimal ..." as true, & I find it very insightful. But it had never occurred to me before seeing it here. $\endgroup$ – gung - Reinstate Monica Sep 14 '17 at 19:15
  • 1
    $\begingroup$ As a principle I figure that if I have answered a question, then there must be something about it that made the answer worthwhile (even if at the time I hate the question itself). On very rare occasions it has taken so much work to figure out what the question meant in the first place that I have not upvoted it; but I cannot imagine any rational consistent set of principles that would allow me to downvote the question as "unclear or not useful"! I suppose a downvote could signal "lack of research," but if that's the case I probably shouldn't be wasting my time with an answer... . $\endgroup$ – whuber Sep 14 '17 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.