Multiple comparisons in a non-parametric test

Question

I am working with a data set that is non-parametric and has 12 treatments. I performed the Kruskal-Wallis test and got a significant $p$-value, and now I would like to conduct a multiple comparisons procedure to see which of the treatments differ significantly. There is a lot of information regarding this topic, but I haven't found anything that specifically addresses this issue. Any ideas??

Answer 1

You are looking for Dunn's test (or, say, the Conover-Iman test). This is very much like a set of pairwise rank sum tests, but Dunn's versions (1) accounts for the pooled variance implied by the null hypothesis, and (2) retains the ranking used to conduct the Kruskal-Wallis test. Performing garden variety Wilcoxon/Mann-Whitney rank sum tests ignores with these issues. One can, of course, perform family-wise error rate or false discovery rate corrections for multiple comparisons with Dunn's test.

Dunn's test is implemented for Stata in the package dunntest (from Stata type net describe dunntest, from(https://alexisdinno.com/stata) whilst connected to the Internet), and for R in the package dunn.test; both packages includes many multiple comparison's adjustment options. One might also perform Dunn's test in SAS using Elliott and Hynan's macro, KW_MC.

As I wrote in a related CV question: there are a few less well-known post hoc pairwise tests to follow a rejected Kruskal-Wallis, including Conover-Iman (like Dunn, but based on the t distribution, rather than the z distribution, strictly more powerful than Dunn's test, and also implemented for Stata in the package conovertest, and for R in the conover.test package), and the Dwass-Steel-Citchlow-Fligner tests.

References
Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3):241–252.