3
$\begingroup$

I want to group my dataset using clustering technique. I apply k-means and used Dunn index and Silhouette Coefficient for the validation (selection of the best number of clusters). Now I want to know what should be the optimal cluster number based on the Dunn index. For your reference i am uploading the DI and SC plot ("cluster size" is the number of clusters). 

The point is that if I keep on increasing the cluster no's the DI value is getting higher after 7. The minimum value in figure is 5. So can't we take 5 as the min no of cluster possible. 

Please suggest what should be the cluster size i need to consider for this plot.

enter image description here

enter image description here

$\endgroup$
  • $\begingroup$ I reckon to go with 7 clusters? en.wikipedia.org/wiki/Dunn_index#Explanation $\endgroup$ – Jan Sila Sep 6 '16 at 9:48
  • $\begingroup$ cluster size What's that? Are you speaking about the number of clusters? $\endgroup$ – ttnphns Sep 6 '16 at 10:09
  • $\begingroup$ Yes. I want to find the optimal no of clusters. $\endgroup$ – Chandra Prakash Sep 6 '16 at 11:05
  • 2
    $\begingroup$ There exist plenty, plenty internal clustering criterions (validation criterions). Each one has its "preferences" or "biases". Dunn's index and Silhouette index are just two of the many. By the way, both these two exist in original as well as in modified versions. The two are quite different conceptually and don't have to be concordant most of the time. In your example, Dunn suggests the 7 clusters and Silhouette - the 2 clusters. If you want a forced negotiation solution you might take the 3-cluster solution as not bad. $\endgroup$ – ttnphns Sep 6 '16 at 11:26
1
$\begingroup$

Indexes such as Dunn's and Silhouette only are a heuristic. Even if they would agree on a single k, you never rely on them anyway. Because they will sometimes not work.

In particular, when the data is not well preprocessed, or the clusters do not match the k-means assumptions, then the indexes will not work either.

It's okay to use the indexes to find the first k to try, though. But I strongly recommend to always 1. visualize the clusters. 2. carefully study the resulting clusters, don't just assume they are good.

$\endgroup$
  • $\begingroup$ But I strongly recommend to always 1. visualize the clusters. 2. carefully study the resulting clusters, don't just assume they are good. So do I. Mind, however, that eye is just one among "clustering criterions" with its own prejudices, so visualization, even if easily possible, isn't the ultimate judge. Some reasonable good and interpretable cluster solutions may be perceived by eye as inconvincing. $\endgroup$ – ttnphns Sep 6 '16 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.