I have some non-Gaussian distributed variable and I need to check if there are significant differences between the values of this variable in 5 different groups.

I have performed Kruskal-Wallis one-way analysis of variance (which came up significant) and after that I had to check which groups were significantly different. Since the groups are kind of sorted (the values of the variable in the first group are supposed to be lower than the values of the variable in the second group which are supposed to be lower than the values of the variable in the third group and so on) I only performed 4 tests:

Group 1 vs Group 2
Group 2 vs Group 3
Group 3 vs Group 4
Group 4 vs Group 5

I have performed this analysis with two different methods. I started by using Dunn's Multiple Comparison Test but nothing came up significant. On the other hand if I use Mann-Whitney test and correct for the number of tests (4) using Bonferroni, 3 tests come up significant.

What does it mean? Which results should I trust?

  • 1
    $\begingroup$ Note that if you a priori expect Group 1 values to be the lowest and Group 5 the highest, then comparing Group 1 with Group 5 will have the highest power to detect a difference. $\endgroup$
    – amoeba
    Commented Aug 15, 2014 at 15:40
  • $\begingroup$ On the Bonferroni correction, you must divide the p value by the number of groups, not the number of tests you performed. $\endgroup$
    – Caramba
    Commented Apr 21, 2015 at 4:43
  • 1
    $\begingroup$ If your alternative is ordered, it would seem better to use a test designed for that situation. $\endgroup$
    – Glen_b
    Commented Sep 24, 2015 at 18:22
  • 3
    $\begingroup$ @Caramba that is incorrect. If you have four groups and do all paired comparisons, you adjust for all six comparisons. $\endgroup$
    – Dave
    Commented Aug 16, 2020 at 12:29

1 Answer 1


You should use a proper post hoc pairwise test like Dunn's test.*

If one proceeds by moving from a rejection of Kruskal-Wallis to performing ordinary pair-wise rank sum tests (with or without multiple comparison adjustments), one runs into two problems:

  1. the ranks that the pair-wise rank sum tests use are not the ranks used by the Kruskal-Wallis test (i.e. you are, in effect, pretending to perform post hoc tests, but are actually using different data than was used in the Kruskal-Wallis test to do so); and

  2. Dunn's test preserves a pooled variance for the tests implied by the Kruskal-Wallis null hypothesis.

Of course, as with any omnibus test (e.g., ANOVA, Cochran's $Q$, etc.), post hoc tests following rejection of a Kruskal-Wallis test which have been adjusted for multiple comparisons may fail to reject all pairwise tests for a given family-wise error rate or given false discovery rate corresponding to a given $\alpha$ for the omnibus test.

* Dunn's test is implemented in Stata in the dunntest package (within Stata type net describe dunntest, from(https://alexisdinno.com/stata)), and in R in the dunn.test package. Caveat: there are a few less well-known post hoc pair-wise tests to follow a rejected Kruskal-Wallis, including Conover-Iman (like Dunn, but based on the t distribution, rather than the z distribution, and strictly more powerful as a post hoc test) which is implemented for Stata in the conovertest package (within Stata type net describe conovertest, from(https://alexisdinno.com/stata)), and for R in the conover.test package, and the Dwass-Steel-Critchlow-Fligner tests.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.