19
$\begingroup$

I have some non-Gaussian distributed variable and I need to check if there are significant differences between the values of this variable in 5 different groups.

I have performed Kruskal-Wallis one-way analysis of variance (which came up significant) and after that I had to check which groups were significantly different. Since the groups are kind of sorted (the values of the variable in the first group are supposed to be lower than the values of the variable in the second group which are supposed to be lower than the values of the variable in the third group and so on) I only performed 4 tests:

Group 1 vs Group 2
Group 2 vs Group 3
Group 3 vs Group 4
Group 4 vs Group 5

I have performed this analysis with two different methods. I started by using Dunn's Multiple Comparison Test but nothing came up significant. On the other hand if I use Mann-Whitney test and correct for the number of tests (4) using Bonferroni, 3 tests come up significant.

What does it mean? Which results should I trust?

Improve this question
$\endgroup$
  • 1
    $\begingroup$ Note that if you a priori expect Group 1 values to be the lowest and Group 5 the highest, then comparing Group 1 with Group 5 will have the highest power to detect a difference. $\endgroup$ – amoeba Aug 15 '14 at 15:40
  • $\begingroup$ On the Bonferroni correction, you must divide the p value by the number of groups, not the number of tests you performed. $\endgroup$ – Caramba Apr 21 '15 at 4:43
  • 1
    $\begingroup$ If your alternative is ordered, it would seem better to use a test designed for that situation. $\endgroup$ – Glen_b Sep 24 '15 at 18:22
  • $\begingroup$ @Caramba that is incorrect. If you have four groups and do all paired comparisons, you adjust for all six comparisons. $\endgroup$ – Dave Aug 16 '20 at 12:29
23
$\begingroup$

You should use Dunn's test$^{*}$. If one proceeds by moving from a rejection of Kruskal-Wallis to performing ordinary pair-wise rank sum tests (with or without multiple comparison adjustments), one runs into two problems: (1) the ranks that the pair-wise rank sum tests use are not the ranks used by the Kruskal-Wallis test; and (2) Dunn's test preserves a pooled variance for the tests implied by the Kruskal-Wallis null hypothesis.

Of course, as with any omnibus test (e.g. ANOVA, Cochran's Q, etc.), post hoc tests following rejection of a Kruskal-Wallis test which have been adjusted for multiple comparisons may fail to reject all pairwise tests for a given family-wise error rate or given false discovery rate corresponding to a given $\alpha$ for the omnibus test.


$^{*}$ Dunn's test is implemented in Stata in the dunntest package (within Stata type net describe dunntest, from(https://alexisdinno.com/stata)), and in R in the dunn.test package. Caveat: there are a few less well-known post hoc pair-wise tests to follow a rejected Kruskal-Wallis, including Conover-Iman (like Dunn, but based on the t distribution, rather than the z distribution) which is implemented for Stata in the conovertest package (within Stata type net describe conovertest, from(https://alexisdinno.com/stata)), and for R in the conover.test package, and the Dwass-Steel-Critchlow-Fligner tests.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.