I used K-means to cluster a large data set that has millions of samples. I tried to create the clusters with different sets of attributes, which, as a result, generated different optimal number of clusters. For example, using attributes A,B,C,D, 5 clusters were created while using attributes X,Y,Z, 4 clusters were created.

My questions are:

  1. How to compare and choose between these two clustering results considering they have different number of clusters and were created with different attributes?
  2. Is there a good metric to use?
  3. Any suggestion for R package that works well for the large data set?
  • 1
    $\begingroup$ Regarding specifically internal clustering validation indices - I would suggest you to read introductory paragraphs (about when we may use them) in my document. Find "Clustering criterions" on my web-page. $\endgroup$ – ttnphns Jun 14 '18 at 18:48

There is a compare() function in the igraph package. Please see the compare() function R documentation here. It would be better to read the papers cited in the documentation.

Other than that I found these discussions helpful: discussion 1, discussion 2, intuition behind Variation of Information.

Hope that these are helpful!

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.