Appropriateness of ANOVA after k-means cluster analysis

Question

The notification after the ANOVA table after K-means analysis indicates that significance levels should not be looked at as the test of equal means, as the cluster solution has been derived based on Euclidean distance to maximize the distance. What test should I use to show whether the means of the clustering variables differ among the clusters? I have seen this warning in k-means outputs' provided ANOVA table, but in some references I see that post-hoc ANOVA tests are run. Should I ignore k-mean ANOVA outputs and run one-way ANOVA with post-hoc tests and interpret them in a traditional way? Or can I only imply about magnitude of F value and which variables contributed more to difference? Another confusion is that clustering variables are not normally distributed violating assumption of ANOVA, then I could use Kruskal-Wallis non-parametric test, but it has assumption about the same distributions. The inter-cluster distributions for the specific variables do not seem the same, some are positively skewed, some are negatively... I have 1275 large sample, 5 clusters, 10 clustering variables measured in PCA scores.

Answer 1

No!

You must not use the same data to 1) perform clustering and 2) hunt for significant differences between the points in the clusters. Even if there's no actual structure in the data, the clustering will impose one by grouping together points which are nearby. This shrinks the within-group variance and grows the across-group variance, which biases you towards false positives.

This effect is surprisingly strong. Here are the results of a simulation that draws a 1000 data points from a standard normal distribution. If we assign the points to one of five groups at random before running the ANOVA, we find that the p-values are uniformly distributed: 5% of the runs are significant at the (uncorrected) 0.05 level, 1% at the 0.01 level, etc. In other words, there is no effect. However, if $k$-means is used to cluster the data into 5 groups, we find a significant effect virtually every time, even though the data has no actual structure.

There is nothing special about a k-means or an ANOVA here--you would see similar effects using non-parametric tests or logistic regression and a decision tree, even just taking the min/max.

After you impose some kind of structure on the data, you cannot to test whether some structure exists, since it obvious does!. As a result, validating clustering algorithms' performance is tricky, particularly if the data are not labelled. However, there are a few approaches to "internal validation", or measuring the clusters' quality without using external data sources. They generally focus on the compactness and separability of the clusters. This review by Lui et al. (2010) might be a good place to start.

Answer 2

Your real problem is data snooping. You can't apply ANOVA or KW if the observations were assigned to groups (clusters) based on the input data set itself. What you can do is to use something like Gap statistic to estimate the number of clusters.

On the other hand, the snooped p-values are biased downward, so if ANOVA or KW test result is insignificant, then the "true" p-value is even larger and you may decide to merge the clusters.

Answer 3

I think you could apply such an approach (i.e. using the statistics, such as F-statistics or t-statistics or whatever), if you toss out the usual null distributions.

What you'd need to do is simulate from the situation in which your null is true, apply the whole procedure (clustering, etc), and then calculate whichever statistic each time. Applied over many simulations, you would get a distribution for the statistic under the null against which your sample value could be compared. By incorporating the data-snooping into the calculation you account for its effect.

[Alternatively one could perhaps develop a resampling-based test (whether based on permutation/randomization or bootstrapping).]

Answer 4

Not exactly an answer, but a proposal on how one would find the solution.

I was thinking about that cluster problem. The test would require sampling from the full dataset and deriving kmeans and seeing if the same kmeans occurs within a distribution (example with clustergram) from various samples (normally kmeans itself produces different kmeans depending on it's starting point. An algorithm that hones in on the same kmeans over multiple iterations like clustergram might be more apt). Just as a mean is derived from samples in statistics. But k means has various proportion sizes for its clusters but the point is do the same means appear within a distribution.

But how would one compare the distributions of the various vars in the cluster? Normally coefficients are derived from a covariance matrix (or predictor matrix?) which is based on a given y. This has no y. So im wondering if each cluster could be whitened using zca (or even pca). Something w eigenvalues. Use this to derive some type of meaningful means or coefficients. Else one has a set of means. Then one needs to derive the standard error. The standard error is definately based on the covariance matrix (again substitute pca or zca)? But I'd have to brush up on standard error. I believe standard error is a function of standard deviation but instead of a variance of a sample its a variance of a mean

Edit:

For statistical significance

Use the gap statistic method as discussed here http://www.datanovia.com/en/lessons/determining-the-optimal-number-of-clusters-3-must-know-methods/#at_pco=wnm-1.0&at_si=609664423560aa01&at_ab=per-2&at_pos=0&at_tot=1

I also recommend this article for a discussion of other related measures https://medium.com/@haataa/how-to-measure-clustering-performances-when-there-are-no-ground-truth-db027e9a871c