5
$\begingroup$

I used K-means to cluster a large data set that has millions of samples. I tried to create the clusters with different sets of attributes, which, as a result, generated different optimal number of clusters. For example, using attributes A,B,C,D, 5 clusters were created while using attributes X,Y,Z, 4 clusters were created.

My questions are:

  1. How to compare and choose between these two clustering results considering they have different number of clusters and were created with different attributes?
  2. Is there a good metric to use?
  3. Any suggestion for R package that works well for the large data set?
Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Regarding specifically internal clustering validation indices - I would suggest you to read introductory paragraphs (about when we may use them) in my document. Find "Clustering criterions" on my web-page. $\endgroup$
    – ttnphns
    Commented Jun 14, 2018 at 18:48
  • $\begingroup$ Partly, the document is copied as this answer. $\endgroup$
    – ttnphns
    Commented Sep 9, 2020 at 21:41
  • $\begingroup$ I would suggest silhouette analysis. rdocumentation.org/packages/cluster/versions/2.1.3/topics/…. $\endgroup$
    – user326964
    Commented Jul 26, 2022 at 14:09
  • $\begingroup$ Measures to compare between clustering partitions aka External clustering criteria $\endgroup$
    – ttnphns
    Commented Jul 23, 2023 at 20:51

2 Answers 2

Reset to default
0
$\begingroup$

There is a compare() function in the igraph package. Please see the compare() function R documentation here. It would be better to read the papers cited in the documentation.

Other than that I found these discussions helpful: discussion 1, discussion 2, intuition behind Variation of Information.

Hope that these are helpful!

Improve this answer
$\endgroup$
0
$\begingroup$

Focusing on a part of question 1: "How to (...) choose between these two clustering results considering they have different number of clusters and were created with different attributes?"

You are asking this question in an abstract manner without reference to the meaning of the data, however in general it is absolutely possible that you have two different absolutely legitimate clusterings of the same objects on two different sets of variables. Obviously, if you have a questionnaire that asks questions about attitudes to the health system and the tax system (say) people may cluster differently regarding health attitudes and regarding tax attitudes and there's nothing wrong with that. Do you really need to choose between them if both are legitimate? If you have to, look at the meaning of the involved variables and ask yourself, a clustering based on which variables is more useful for what you want to use it for? The data cannot decide this.

That said, data analytically these clusterings may be of different quality. If the number of involved variables is reasonably low, you can look at pairs plots of density levels, and you can also resample small data subsets and see whether the clusterings you get on them correspond in a stable manner to the overall clusterings.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.