The notification after the ANOVA table after K-means analysis indicates that significance levels should not be looked at as the test of equal means, as the cluster solution has been derived based on Euclidean distance to maximize the distance. What test should I use to show whether the means of the clustering variables differ among the clusters? I have seen this warning in k-means outputs' provided ANOVA table, but in some references I see that post-hoc ANOVA tests are run. Should I ignore k-mean ANOVA outputs and run one-way ANOVA with post-hoc tests and interpret them in a traditional way? Or can I only imply about magnitude of F value and which variables contributed more to difference? Another confusion is that clustering variables are not normally distributed violating assumption of ANOVA, then I could use Kruskal-Wallis non-parametric test, but it has assumption about the same distributions. The inter-cluster distributions for the specific variables do not seem the same, some are positively skewed, some are negatively... I have 1275 large sample, 5 clusters, 10 clustering variables measured in PCA scores.

  • $\begingroup$ Why do you need to test the equality of means? Can't you just test how your model works out of sample? $\endgroup$
    – James
    Sep 22 '14 at 14:24
  • $\begingroup$ I wanted to determine which variables' means differ among clusters, i.e. whether mean of v1 in cluster1 is different from mean of v1 in cluster, 2, 3, 4, 5. I can of course see that by making a graph, but it does not tell about the statistical difference. Test for statistical difference made me confused, as for ANOVA my data did not meet normal distribution assumption, but for Kruskal Wallis test same shape distribution assumption among cluster groups. $\endgroup$
    – Inga
    Sep 22 '14 at 15:06
  • 1
    $\begingroup$ As @James pointed in his answer, you are "snooping". What could be a reason to test for significance between groups which you (your clustering) preselected to differ as much as possible? Here is no any mark of random or proportional sampling from populations which are distinct on the basis of some external, backgroud characteristics. $\endgroup$
    – ttnphns
    Sep 22 '14 at 18:53
  • $\begingroup$ Thank you for the answers! My confusion appeared as in some sources I see that statistical mean comparisons are not appropriate in this situation as you have also pointed,but e.g a quote from 1 book's chapter indicates opposite: "we usually examine the means for each cluster on each dimension using ANOVA to assess how distinct our clusters are. Ideally, we would obtain significantly different means for most, if not all dimensions, used in the analysis. The magnitude of the F values performed on each dimension is an indication of how well the respective dimension discriminates between clusters" $\endgroup$
    – Inga
    Sep 22 '14 at 19:08
  • 2
    $\begingroup$ You have the right to assess differences between clusters by the characteristics used to cluster - in order to find out the most discriminative ones. When doing so, you may compute relative differences, F's, and even p-values. As indicators of effect size. Not as indicators of statistical significance (which refer to populations). $\endgroup$
    – ttnphns
    Sep 22 '14 at 20:23


You must not use the same data to 1) perform clustering and 2) hunt for significant differences between the points in the clusters. Even if there's no actual structure in the data, the clustering will impose one by grouping together points which are nearby. This shrinks the within-group variance and grows the across-group variance, which biases you towards false positives.

This effect is surprisingly strong. Here are the results of a simulation that draws a 1000 data points from a standard normal distribution. If we assign the points to one of five groups at random before running the ANOVA, we find that the p-values are uniformly distributed: 5% of the runs are significant at the (uncorrected) 0.05 level, 1% at the 0.01 level, etc. In other words, there is no effect. However, if $k$-means is used to cluster the data into 5 groups, we find a significant effect virtually every time, even though the data has no actual structure.

Simulation results showing a uniform distribution of pvalues for the random assignments and a highly skewed (almost all 0.05 or less) distribution of p values after clustering

There is nothing special about a k-means or an ANOVA here--you would see similar effects using non-parametric tests or logistic regression and a decision tree, even just taking the min/max.

After you impose some kind of structure on the data, you cannot to test whether some structure exists, since it obvious does!. As a result, validating clustering algorithms' performance is tricky, particularly if the data are not labelled. However, there are a few approaches to "internal validation", or measuring the clusters' quality without using external data sources. They generally focus on the compactness and separability of the clusters. This review by Lui et al. (2010) might be a good place to start.


Your real problem is data snooping. You can't apply ANOVA or KW if the observations were assigned to groups (clusters) based on the input data set itself. What you can do is to use something like Gap statistic to estimate the number of clusters.

On the other hand, the snooped p-values are biased downward, so if ANOVA or KW test result is insignificant, then the "true" p-value is even larger and you may decide to merge the clusters.


I think you could apply such an approach (i.e. using the statistics, such as F-statistics or t-statistics or whatever), if you toss out the usual null distributions.

What you'd need to do is simulate from the situation in which your null is true, apply the whole procedure (clustering, etc), and then calculate whichever statistic each time. Applied over many simulations, you would get a distribution for the statistic under the null against which your sample value could be compared. By incorporating the data-snooping into the calculation you account for its effect.

[Alternatively one could perhaps develop a resampling-based test (whether based on permutation/randomization or bootstrapping).]

  • 3
    $\begingroup$ Right, that's the idea behind the Gap statistic. $\endgroup$
    – James
    Sep 24 '14 at 2:06

Not exactly an answer, but a proposal on how one would find the solution.

I was thinking about that cluster problem. The test would require sampling from the full dataset and deriving kmeans and seeing if the same kmeans occurs within a distribution (example with clustergram) from various samples (normally kmeans itself produces different kmeans depending on it's starting point. An algorithm that hones in on the same kmeans over multiple iterations like clustergram might be more apt). Just as a mean is derived from samples in statistics. But k means has various proportion sizes for its clusters but the point is do the same means appear within a distribution.

But how would one compare the distributions of the various vars in the cluster? Normally coefficients are derived from a covariance matrix (or predictor matrix?) which is based on a given y. This has no y. So im wondering if each cluster could be whitened using zca (or even pca). Something w eigenvalues. Use this to derive some type of meaningful means or coefficients. Else one has a set of means. Then one needs to derive the standard error. The standard error is definately based on the covariance matrix (again substitute pca or zca)? But I'd have to brush up on standard error. I believe standard error is a function of standard deviation but instead of a variance of a sample its a variance of a mean


For statistical significance

Use the gap statistic method as discussed here http://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/#at_pco=wnm-1.0&at_si=609664423560aa01&at_ab=per-2&at_pos=0&at_tot=1

I also recommend this article for a discussion of other related measures https://medium.com/@haataa/how-to-measure-clustering-performances-when-there-are-no-ground-truth-db027e9a871c


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.