Strange result of post-hoc test

Question

I have data for a test on three groups. The measured variable is ratio scaled. The R code is

g1a<-c(7, 3, 40)
g2a<-c(1,1,2)
g3a<-c(0,0,0)

Since the sample is small and normality cannot be guaranteed, I run a Kruskal Wallis test to check for significance:

l<-list(g1a,g2a,g3a)
kruskal.test(l)

The p-value is 0.02336, which is nice.

Now I run a post-hoc test, using the Mann-Whitney U:

wilcox.test(g1a,g2a,paired=FALSE,exact=TRUE)
wilcox.test(g2a,g3a,paired=FALSE,exact=TRUE)
wilcox.test(g1a,g3a,paired=FALSE,exact=TRUE)

All the resulting p-values are above 0.05 (0.07652, 0.0636, 0.05935). This is very strange. Shouldn't one of these tests give a much lower p-value? Especially since I'd have to use some sort of correction to account for the multiple comparisons in the post-hoc test. In other words: how can I interpret this result?

Answer 1

Think of it this way - overall, there's a significant difference, but it's a little hard to say exactly which two are significantly different. Alternatively, consider the chances of having three p-values less than 0.1 (even though they aren't independent of each other) - pretty small, right? So, again overall, we might suspect something significant is in the data, without being able to tell exactly where.

Your small sample sizes don't help; they mean the powers of your tests are very low, and also severely constrain what sort of p-values you can get, as the following example shows:

> g1a <- rnorm(3,0,1)
> g2a <- rnorm(3,2.5,1)
> g3a <- rnorm(3,5,1)
> 
> y <- list(g1a,g2a,g3a)
> y
[[1]]
[1] -2.31356435 -0.09903136 -0.42037052

[[2]]
[1] 2.806082 2.799857 3.383844

[[3]]
[1] 6.543636 6.845559 4.838341

> kruskal.test(y)

    Kruskal-Wallis rank sum test

data:  y 
Kruskal-Wallis chi-squared = 7.2, df = 2, p-value = 0.02732

So far, so good. On to the three Wilcoxon tests:

> wilcox.test(g1a,g2a,paired=FALSE,exact=TRUE)

    Wilcoxon rank sum test

data:  g1a and g2a 
W = 0, p-value = 0.1
alternative hypothesis: true location shift is not equal to 0 

> wilcox.test(g2a,g3a,paired=FALSE,exact=TRUE)

    Wilcoxon rank sum test

data:  g2a and g3a 
W = 0, p-value = 0.1
alternative hypothesis: true location shift is not equal to 0 

> wilcox.test(g1a,g3a,paired=FALSE,exact=TRUE)

    Wilcoxon rank sum test

data:  g1a and g3a 
W = 0, p-value = 0.1
alternative hypothesis: true location shift is not equal to 0 

All three p-values at 0.1, but we can't get more extreme - W = 0 - so evidently we've hit a sample size imposed limit on p-values.

Answer 2

Your mistake is in choosing the Wilcoxon/Mann-Whitney rank-sum tests as your post hoc tests following the rejection of the Kruskal-Wallis. The appropriate pos hoc test is Dunn's test^* which properly (1) accounts for pooled variance assumed by the null hypothesis, and (2) uses the same ranks for your data as used in the construction of the Kruskal-Wallis test. The vanilla rank-sum tests entail separate estimates of variance for each pair-wise test, and ignore the rankings of the total data set as performed with a Kruskal-Wallis test.

Dunn's test is implemented for Stata in the dunntest package (within Stata type net describe dunntest, from(https://alexisdinno.com/stata)), and for R in the dunn.test package. Not sure about implementations in SAS.

Reference

Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3):241–252.

^* There are some far less used alternatives to Dunn's test including the Conover-Iman (like Dunn, but based on the t distribution, rather than the z distribution, implemented for Stata in the conovertest package, and for R in the conover.test package), and the Dwass-Steel-Citchlow-Fligner tests.

The second issue, which arises even when using appropriate tests, is that you are making the false assumption that rejection of an omnibus null hypothesis means there must be at least one rejection of pairwise post hoc null hypothesis.

Answer 3

The same thing can happen with the ANOVA test when normal distributions can be assumed. The differences beteween the three is apparently just large enough to see that they are different but not quite large enough to distinguish the difference between pairs. Note the overall p-value is a little less than 0.05 and each of the pairwise tests are all slightly larger than 0.05. With a larger sample size you might find that each one is different from the other two. But the inference here is that the medians differ but you aren't sure which pair(s) you can attribute this to.

Answer 4

This is a well-known problem in two-stage comparisons, observed e.g., already by Gabriel [Gabriel KR (1969) Simultaneous test procedures - some theory of multiple comparisons. The Annals Mathematical Statistics 40(1):224-250].