Non-significant Kruskal-Wallis and significant post hoc tests

Question

I have done this analysis where in Figure A-D. the Kruskal-Wallis global p-value is non-significant, but the p-value for pair CS and SCS in Figure A however is significant (at P<0.05). I read somewhere that if the global p-value is non-significant we don't report post-hoc pairwise tests. So In figure A, B, C, D and H, should I just delete the p-values for paired tests as the global p-values in these plots are not significant? Could someone please clarify this?

Answer 1

In general, if you use an omnibus test, such as an ANOVA F-test or a Kruskal-Wallis H-test, it is illogical and poor practice to conduct pairwise comparisons when you fail to reject the null hypothesis on the omnibus test. Conducting the comparisons flies in the face of the omnibus: insufficient evidence to conclude differences does not warrant further investigation, as a general rule.

Usually, I would say report analyses you run, but in this case (which is different from selective reporting), the post-hoc p-values are inappropriate to interpret and should be omitted. The omnibus p-value is appropriate since this is the “gatekeeper” test.

Answer 2

Illustrating @LSC's answer (+1), here is an example to show that doing ad hoc two-sample Wilcoxon tests can lead to 'false discoveries' if a Kruskal-Wallis test is not significant.

Suppose we have five groups of size $n = 20$ with values distributed uniformly on the same interval. So, ideally we should not find any significant differences at all.

Let the significance level of all tests be 10%. Then the K-W test will reject $H_0$ that all five groups come from the same population distribution about 10% of the time.

If we were to do ad hoc two sample Wilcoxon tests to compare the two groups out of five that have the most different medians, we would reject the null hypothesis that those two groups differ another 10% of the time.

A simulation in R of 10,000 such datasets, each with five groups of 20 observations, is shown below:

set.seed(2019)
m = 10^4;  p.kw = p.wx = numeric(m)
for(i in 1:m) {
  x1 = runif(20, 10, 25);  x2 = runif(20, 10, 25)
  x3 = runif(20, 10, 25);  x4 = runif(20, 10, 25)
  x5 = runif(20, 10, 25)
  p.kw[i] = kruskal.test(list(x1,x2,x3,x4,x5))$p.val
  MAT=rbind(x1,x2,x3,x4,x5)
  h = apply(MAT,1,median)
  mx = which(h==max(h));  mn = which(h==min(h))
  p.wx[i] = wilcox.test(MAT[mx,1],MAT[mn,])$p.val
}
mean(p.kw<.1);  mean(p.wx<.1)
[1] 0.0947      # rejection rate for K-W
[1] 0.1017      # rej rate: all Wilcoxon for two extremes
mean(p.wx[p.kw > .1] < .1)
[1] 0.09632166  # rej rate: improper ad hoc Wilcoxon tests

It happens that one of the unwarranted ad hoc Wilcoxon rank sum tests occurred for the last of the 10,000 iterations. Boxplots for the five groups are shown, followed by a non-significant (10% level) K-W test and a significant two-sample Wilcoxon test comparins groups 2 and 3.

kruskal.test(list(x1,x2,x3,x4,x5))$p.value
[1] 0.3357694             # not significant at 10%
wilcox.test(x2,x3)$p.val
[1] 0.09108615            # 'significant' improper ad hoc test

Added per Comment: The P-value for a K-W test comparing the same two groups gives very nearly the same P-value as the Wilcoxon test above; both P-values essentially 0.9.

kruskal.test(list(x2,x3))$p.val
[1] 0.08835202

Answer 3

There is a question underlying OP's example that is not a trivial one: If a test like KW assesses a hypothesis that all groups are (stochastically) equal, and if we find a non-significant result for KW, but then a significant result for pairwise tests, How is it that these tests are valid? Why would we ever want to use an omnibus test instead of jumping to pairwise tests?

Some of the comments on the original post partially address this situation.

One issue is that KW looks at the data in all groups simultaneously, whereas presumably the software in the OP's example is using pairwise Mann-Whitney tests which look at two groups at a time while ignoring the other groups. So, simply, they are addressing the analysis in different ways. Usually it is considered better to look at all groups simultaneously. This can also avoid potential incommensurate pairwise results (where e.g. A > B, B > C, and C > A). As mentioned in the comments on the original post, we might use Dunn's test (1964) or Conover-Iman instead of a simple pairwise approach.

Likely another issue here is that the post-hoc approach used may not have adjusted p values for multiple tests. If this is the case, then the likelihood of finding a significant result in the pairwise tests, just by chance, is elevated.

I also think there's some sense to the idea that if are really interested in comparing groups, that we can skip the omnibus test and look at group comparisons (Link1, Link2). In the context of KW, this logic may also apply to effect size statistics. That is, using pairwise or maximum Vargha and Delaney's A rather than epsilon-squared.