# Strange result of post-hoc test

I have data for a test on three groups. The measured variable is ratio scaled. The R code is

g1a<-c(7, 3, 40)
g2a<-c(1,1,2)
g3a<-c(0,0,0)


Since the sample is small and normality cannot be guaranteed, I run a Kruskal Wallis test to check for significance:

l<-list(g1a,g2a,g3a)
kruskal.test(l)


The p-value is 0.02336, which is nice.

Now I run a post-hoc test, using the Mann-Whitney U:

wilcox.test(g1a,g2a,paired=FALSE,exact=TRUE)
wilcox.test(g2a,g3a,paired=FALSE,exact=TRUE)
wilcox.test(g1a,g3a,paired=FALSE,exact=TRUE)


All the resulting p-values are above 0.05 (0.07652, 0.0636, 0.05935). This is very strange. Shouldn't one of these tests give a much lower p-value? Especially since I'd have to use some sort of correction to account for the multiple comparisons in the post-hoc test. In other words: how can I interpret this result?

• (+1) Excellent, I was looking for such an example for a long time. This shows the power of simultaneous versus consecutive testing. May 27 '12 at 15:16
• see the answers to this question as well for the relationship between global tests and post-hoc tests.
– jank
Oct 28 '13 at 13:00

Think of it this way - overall, there's a significant difference, but it's a little hard to say exactly which two are significantly different. Alternatively, consider the chances of having three p-values less than 0.1 (even though they aren't independent of each other) - pretty small, right? So, again overall, we might suspect something significant is in the data, without being able to tell exactly where.

Your small sample sizes don't help; they mean the powers of your tests are very low, and also severely constrain what sort of p-values you can get, as the following example shows:

> g1a <- rnorm(3,0,1)
> g2a <- rnorm(3,2.5,1)
> g3a <- rnorm(3,5,1)
>
> y <- list(g1a,g2a,g3a)
> y
[[1]]
[1] -2.31356435 -0.09903136 -0.42037052

[[2]]
[1] 2.806082 2.799857 3.383844

[[3]]
[1] 6.543636 6.845559 4.838341

> kruskal.test(y)

Kruskal-Wallis rank sum test

data:  y
Kruskal-Wallis chi-squared = 7.2, df = 2, p-value = 0.02732


So far, so good. On to the three Wilcoxon tests:

> wilcox.test(g1a,g2a,paired=FALSE,exact=TRUE)

Wilcoxon rank sum test

data:  g1a and g2a
W = 0, p-value = 0.1
alternative hypothesis: true location shift is not equal to 0

> wilcox.test(g2a,g3a,paired=FALSE,exact=TRUE)

Wilcoxon rank sum test

data:  g2a and g3a
W = 0, p-value = 0.1
alternative hypothesis: true location shift is not equal to 0

> wilcox.test(g1a,g3a,paired=FALSE,exact=TRUE)

Wilcoxon rank sum test

data:  g1a and g3a
W = 0, p-value = 0.1
alternative hypothesis: true location shift is not equal to 0


All three p-values at 0.1, but we can't get more extreme - W = 0 - so evidently we've hit a sample size imposed limit on p-values.

• The Wilcoxon/Mann-Whitney test is not the appropriate post hoc test following rejection of the Kruskal-Wallis test. Jun 27 '14 at 18:25
• @Alexis - I never said it was. The point of my answer was that he can't get (at least) one of his p-values from the Wilcoxon tests low because of sample size issues. I read the question as being about not understanding why he got a low Kruskal-Wallis p value but didn't get any low Wilcoxon p-values, not about whether the Wilcoxon was an appropriate post-hoc test or not. Still, a good point that I wish I'd made! (+1). Jun 27 '14 at 20:22

Your mistake is in choosing the Wilcoxon/Mann-Whitney rank-sum tests as your post hoc tests following the rejection of the Kruskal-Wallis. The appropriate pos hoc test is Dunn's test* which properly (1) accounts for pooled variance assumed by the null hypothesis, and (2) uses the same ranks for your data as used in the construction of the Kruskal-Wallis test. The vanilla rank-sum tests entail separate estimates of variance for each pair-wise test, and ignore the rankings of the total data set as performed with a Kruskal-Wallis test.

Dunn's test is implemented for Stata in the dunntest package (within Stata type net describe dunntest, from(https://alexisdinno.com/stata)), and for R in the dunn.test package. Not sure about implementations in SAS.

Reference

Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3):241–252.

* There are some far less used alternatives to Dunn's test including the Conover-Iman (like Dunn, but based on the t distribution, rather than the z distribution, implemented for Stata in the conovertest package, and for R in the conover.test package), and the Dwass-Steel-Citchlow-Fligner tests.

The second issue, which arises even when using appropriate tests, is that you are making the false assumption that rejection of an omnibus null hypothesis means there must be at least one rejection of pairwise post hoc null hypothesis.

• Yes, and in fact in this example, Dunn's test will return a significant result for groups 1 and 3, even with a Bonferroni adjustment. Conover test will also return significant results. I didn't check DSCF. Oct 29 '19 at 13:29
• I'm a bit late to the (my own) party but the choice of Mann-Whitney U as a post-hoc for the Kruskal-Wallis is explicitly stated in "Statistik für Psychologen" by Peter Zofel (p. 146). Not saying it's right because it is written in a book. Just a cautionary tale for anybody considering buying this book.
– xmjx
Nov 9 '19 at 21:49
• @xmjx Zofel is doing his readers a poor service for the two reasons—(1) and (2)—I give in my answer. Nov 9 '19 at 22:07

The same thing can happen with the ANOVA test when normal distributions can be assumed. The differences beteween the three is apparently just large enough to see that they are different but not quite large enough to distinguish the difference between pairs. Note the overall p-value is a little less than 0.05 and each of the pairwise tests are all slightly larger than 0.05. With a larger sample size you might find that each one is different from the other two. But the inference here is that the medians differ but you aren't sure which pair(s) you can attribute this to.

This is a well-known problem in two-stage comparisons, observed e.g., already by Gabriel [Gabriel KR (1969) Simultaneous test procedures - some theory of multiple comparisons. The Annals Mathematical Statistics 40(1):224-250].