I used K-means to cluster a large data set that has millions of samples. I tried to create the clusters with different sets of attributes, which, as a result, generated different optimal number of clusters. For example, using attributes A,B,C,D, 5 clusters were created while using attributes X,Y,Z, 4 clusters were created.
My questions are:
- How to compare and choose between these two clustering results considering they have different number of clusters and were created with different attributes?
- Is there a good metric to use?
- Any suggestion for R package that works well for the large data set?