0
$\begingroup$

I'm working on a problem where observations are being clustered within groups but I'd also like to compare the groups. However I am not sure of the best way to compare the groups.

In total I have about 1,000 groups each containing 5 clusters created using k-means. The clusters are based on anywhere from 100-10,000 data points.

For a single group, I can examine the clusters and calculate the within- and between-cluster variation, as well as the cluster means, etc.

However since I have 1,000 groups I'm trying to find different ways I can summarize these data. For example perhaps plotting the groups' mean within-cluster variation vs. mean between-cluster variation (so there is a single point per group).

Most of my ideas are simply to take the mean of something. Are there any better/other methods for summarizing group clusters?

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ What is the purpose of your analysis? Are you trying to find the group with the greatest inter-group variation? $\endgroup$
    – David G. Stork
    Commented Jan 5, 2016 at 1:00
  • $\begingroup$ @DavidG.Stork the purpose is to reduce the information in each "group" by "prototypes" (i.e. clusters) and to then compare the groups in terms of their prototypes. $\endgroup$
    – Ellis Valentiner
    Commented Jan 5, 2016 at 13:41
  • $\begingroup$ After clustering within a group, you could calculate several cluster quality indicators (silhouette index, Dunn's index, Davies-Bouldin index, Hubert's $\Gamma$), and then compare these across groups. $\endgroup$
    – user32398
    Commented Jan 5, 2016 at 14:24
  • $\begingroup$ @LEP I am thinking that the clustering aspect is less important and the comparison of the groups is more so. Can I just do a one-way anova for each group? $\endgroup$
    – Ellis Valentiner
    Commented Jan 5, 2016 at 15:19
  • $\begingroup$ Sure, one-way ANOVA will work, but recall that will only address one feature across all the groups (univariate). You could test many of the same features across many groups using a Hotelling test (MANOVA). Also, through chance alone, you could get significantly different means, so you'd need to partition to objects in each group into folds (cross-validation) and then keep an average value of e.g. -log(p-value). $\endgroup$
    – user32398
    Commented Jan 5, 2016 at 15:35

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.