Differences between Dwass-Steel-Critchlow-Fligner and Mann-Whitney U-test for a post-hoc pairwise analysis

Question

If you have one control and four groups of patients with a non-normal small sample size, there are differences in these tests for a post-hoc analysis after Kruskal-Wallis?

Answer 1

There are absolutely differences.

The Mann-Whitney U test (i.e. the rank sum test) is not appropriate as a post hoc test following the omnibus Kruskal-Wallis nonparametric analog to the one-way ANOVA for two reasons:

1. The rank sum test uses different ranks than those employed in the Kruskal-Wallis test (i.e. in both tests you mush the observations together, then rank them, then separate the ranks by group—the rank sum ignores the ranks you got with the omnibus test because it only ranks based on two groups only, instead of based on the two groups' ranks out of all $k$ groups).
2. The rank sum test does not use the pooled variance implied by the null hypothesis in the Kruskal-Wallis test (e.g., just as in one-way ANOVA where the post hoc t tests use an estimate of the pooled variance).

Dunn's test was (as far as I know) the first post hoc test for Kruskal-Wallis. It is based on a z approximation to the distribution of a rank sum-like test statistic that addresses both points (1) and (2). The Conover-Iman test is similar to Dunn's test, but is based upon a t distribution, and is strictly more powerful than Dunn's test when rejecting the Kruskal-Wallis null hypothesis.

The Dwass-Steel-Crichtlow-Fligner test also addresses (1) and (2), but has a specific approach to controlling the familywise error rate (FWER) built in. Crichtlow and Fligner interpret Dunn's test as necessarily and exclusively incorporating the Bonferroni adjustment (an incorrect interpretation in my opinion—indeed, I have implemented Dunn's test for Stata and for R to include a wide range of false discovery rate (FDR) and FWER adjustments for multiple comparisons), and have implemented the Conover-Iman test for Stata and for R with the same selection of methods to control the FDR and FWER.

References

Conover, W. J. and Iman, R. L. (1979). On multiple-comparisons procedures. Technical Report LA-7677-MS, Los Alamos Scientific Laboratory.

Crichtlow, D. E. and Fligner, M. A. (1991). On distribution-free multiple comparisons in the one-way analysis of variance. Communications in Statistics—Theory and Methods, 20(1):127.

Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3):241–252.