Using Multiple Comparisons for Kruskal Wallis Follow up analysis

I am performing the Kruskal Wallis test on ~100 samples and wish to group statistically ‘similar’ samples together. The method I am using is described in Andy Field’s DISCOVERING STATISTICS USING IBM SPSS STATISTICS (Section 7.6.2 pg308-9). This method, which Field calls the ‘non-parametric step down procedure’, consists of the following steps 1 Order the samples by the mean sample rank 2 Conduct a KW test on the 1st and 2nd samples of the ordered set 3a If the test passes (the two are statistically similar), perform the KW again on the 1st, 2nd, and 3rd ordered samples 3b If this test rejects, then sample 1 is in its own distinct group. Repeat steps 2 and 3 with samples 2 and 3 4 After repeating these steps through the whole set of samples, I should have distinct subgroups of homogenous samples

I don’t see why this method avoids the problem of inflated family wise error rate, nor am I sure whether this is the correct method for grouping similar samples together. I’ve come across a number of other tests used for multiple sample comparisons (e.g. Dunn’s test) but these perform pairwise comparison tests and doesn’t group the data together in a straight forward manner.

Is the step down method the correct thing to do in my situation? Is it statistically correct? Are there alternative procedures for grouping statistically similar samples together?

(Currently conducting these tests using Python)

• I would have thought the problem which you hint at might be better solved by some form of cluster analysis but it is hard to be sure. – mdewey Jan 8 at 14:10
• Clustering did cross my mind, but its not clear which distance metric to use. One idea was to conduct a number of pairwise tests and try to create a network of connections between the accepted and rejected hypotheses but this still leaves some areas of confusion. e.g. if A is similar to B and C, but B is not similar to C, then how are these three samples grouped? (A,B) and (C), or (A,C) and (B) or (A,B,C)? – dm1029 Jan 9 at 13:32