I am performing the Kruskal Wallis test on ~100 samples and wish to group statistically ‘similar’ samples together. The method I am using is described in Andy Field’s
DISCOVERING STATISTICS USING IBM SPSS STATISTICS (Section 7.6.2 pg308-9). This method, which Field calls the ‘non-parametric step down procedure’, consists of the following steps
1 Order the samples by the mean sample rank
2 Conduct a KW test on the 1st and 2nd samples of the ordered set
3a If the test passes (the two are statistically similar), perform the KW again on the 1st, 2nd, and 3rd ordered samples
3b If this test rejects, then sample 1 is in its own distinct group. Repeat steps 2 and 3 with samples 2 and 3
4 After repeating these steps through the whole set of samples, I should have distinct subgroups of homogenous samples
I don’t see why this method avoids the problem of inflated family wise error rate, nor am I sure whether this is the correct method for grouping similar samples together. I’ve come across a number of other tests used for multiple sample comparisons (e.g. Dunn’s test) but these perform pairwise comparison tests and doesn’t group the data together in a straight forward manner.
Is the step down method the correct thing to do in my situation? Is it statistically correct? Are there alternative procedures for grouping statistically similar samples together?
(Currently conducting these tests using Python)