3
$\begingroup$

I want to group my dataset using clustering technique. I apply k-means and used Dunn index and Silhouette Coefficient for the validation (selection of the best number of clusters). Now I want to know what should be the optimal cluster number based on the Dunn index. For your reference i am uploading the DI and SC plot ("cluster size" is the number of clusters). 

The point is that if I keep on increasing the cluster no's the DI value is getting higher after 7. The minimum value in figure is 5. So can't we take 5 as the min no of cluster possible. 

Please suggest what should be the cluster size i need to consider for this plot.

enter image description here

enter image description here

Improve this question
$\endgroup$
4
  • $\begingroup$ I reckon to go with 7 clusters? en.wikipedia.org/wiki/Dunn_index#Explanation $\endgroup$
    – Jan Sila
    Commented Sep 6, 2016 at 9:48
  • $\begingroup$ cluster size What's that? Are you speaking about the number of clusters? $\endgroup$
    – ttnphns
    Commented Sep 6, 2016 at 10:09
  • $\begingroup$ Yes. I want to find the optimal no of clusters. $\endgroup$
    – Chandra Prakash
    Commented Sep 6, 2016 at 11:05
  • 2
    $\begingroup$ There exist plenty, plenty internal clustering criterions (validation criterions). Each one has its "preferences" or "biases". Dunn's index and Silhouette index are just two of the many. By the way, both these two exist in original as well as in modified versions. The two are quite different conceptually and don't have to be concordant most of the time. In your example, Dunn suggests the 7 clusters and Silhouette - the 2 clusters. If you want a forced negotiation solution you might take the 3-cluster solution as not bad. $\endgroup$
    – ttnphns
    Commented Sep 6, 2016 at 11:26

1 Answer 1

Reset to default
1
$\begingroup$

Indexes such as Dunn's and Silhouette only are a heuristic. Even if they would agree on a single k, you never rely on them anyway. Because they will sometimes not work.

In particular, when the data is not well preprocessed, or the clusters do not match the k-means assumptions, then the indexes will not work either.

It's okay to use the indexes to find the first k to try, though. But I strongly recommend to always 1. visualize the clusters. 2. carefully study the resulting clusters, don't just assume they are good.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ But I strongly recommend to always 1. visualize the clusters. 2. carefully study the resulting clusters, don't just assume they are good. So do I. Mind, however, that eye is just one among "clustering criterions" with its own prejudices, so visualization, even if easily possible, isn't the ultimate judge. Some reasonable good and interpretable cluster solutions may be perceived by eye as inconvincing. $\endgroup$
    – ttnphns
    Commented Sep 6, 2016 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.