14
$\begingroup$

I am surveying the use of statistical significance testing (SST) to validate the results of cluster analysis. I have found several papers around this topic, such as

  • "Statistical Significance of Clustering for High-Dimension, Low–Sample Size Data" by Liu, Yufeng et al. (2008)
  • "On some significance tests in cluster analysis", by Bock (1985)

But I am interested in finding some literature arguing that SST is NOT appropriate to validate results of cluster analysis. The only source I have found claiming this is a web page of a software vendor

To clarify:

I am interested in testing whether a significant cluster structure has been found as a result of cluster analysis, so, I'd like to know of papers supporting or refuting the concern "about the possibility of post-hoc testing of the results of exploratory data analysis used to find clusters".

I've just found a paper from 2003, "Clustering and classification methods" by Milligan and Hirtle saying, for example, that using ANOVA would be an invalid analysis since data have not have random assignments to the groups.

$\endgroup$
  • $\begingroup$ This is a good question, but it may be worth pointing out that it is phrased in a way that makes it seem like there's a dichotomy: either you can test for the significance of clustering or you cannot. The situation is different, though, because "cluster analysis" means various things. In the referenced papers the focus is on testing whether there is evidence for clustering. In the software manual, concern is rightly expressed about the possibility of post-hoc testing of the results of exploratory data analysis used to find clusters. There is no contradiction here. $\endgroup$ – whuber Nov 21 '11 at 14:36
  • $\begingroup$ Thanks for answering. You are right about the way I posed the question. I am interested in testing whether a significant cluster structure has been found as a result of cluster analysis, so, I'd like to know of papers supporting or refuting the concern "about the possibility of post-hoc testing of the results of exploratory data analysis used to find clusters". I've just found a paper from 2003 "Clustering and classification methods" by Milligan and Hirtle saying, for example, that using ANOVA would be an invalid analysis since data have not have random assignments to the groups. $\endgroup$ – DPS Nov 21 '11 at 16:29
  • $\begingroup$ Might help: Blinded by science: the managerial consequences of inadequately validated cluster analysis solutions, mrs.org.uk/ijmr_article/article/78841 $\endgroup$ – rolando2 Dec 9 '15 at 0:46
3
$\begingroup$

It is fairly obvious that you cannot (naively) test for difference in distributions for groups that were defined using the same data. This is known as "selective testing", "double dipping", "circular inference", etc.

An example would be performing a t-test on the heights of "tall" and "short" people in your data. The null will (almost) always be rejected.

Having said that- one may indeed account for the clustering stage at the testing stage. I am unfamiliar, however, with a particular reference that does that, but I suspect this should have been done.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I agree on that the null will almost always be rejected when applying a significance test on the different cluster groups. Though – this should only be the case if the clustering actually managed to nicely separate groups for all variables considered in the CA? Couldn’t one use a significance test to determine if there are variables which are not separated well between the groups (meaning apply a test for each variable)? Could you please elaborate on the statistical reason why this is not recommended/sensible? $\endgroup$ – luke Jul 27 '16 at 13:45
  • $\begingroup$ The formal argument is that the error term of each measurement is not centered about zero. Think of my tall/short example: all the people are drawn from the same distribution, but the "tall" group has errors with positive mean, and the "short" negative mean. $\endgroup$ – JohnRos Jul 27 '16 at 17:47
0
$\begingroup$

Instead of hypothesis testing with a given test, I would recommend bootstrapping means or other summary estimates between clusters. For instance you could rely on percentile bootstrap with at least 1000 samples. The key point is to apply clustering independently to each bootstrap sample.

This approach would be quite robust, provide evidence for differences, and support your claim of significant between-cluster difference. In addition, you could generate another variable (say between-cluster difference) and bootstrap estimates of such difference variable would be similar to a formal test of hypothesis.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.