# Mann-Whitney (2 groups) contradicted by Kruskal-Wallis (3 groups)

We have two groups that are significantly different (tested with Mann-Whitney). When a third group is added, the Kruskal-Wallis tests provides a non-significant result.

This implies that two or more groups were not significantly different from each other; hence the non-significant result.

or

Does this imply that we do not have enough power to detect any effect (type-1 error) and the third group is likely not different from the other two groups? So the result is inconclusive.

• Let's flag the elementary but fundamental principle: Plot the data to see what is going on. Further: why not show us the data? Commented Apr 6, 2019 at 8:15
• There are multiple possible explanations; we can't guess which one(s) will be the case for you. (Most of the possible explanations are explored in answers to other questions already on site) Commented Apr 6, 2019 at 8:29
• You are using the incorrect post hoc test. Mann-Whitney (a) does not use the same rankings as the K-S, so no surprise you are getting strange results, and (b) does not use the pooled variance assumed by the K-S null. Try Dunn's test or, even better, the Conover-Iman test. Commented Apr 15, 2023 at 20:10

One possibility is that the third sample introduces extra variability or violates assumptions for the Kruskal-Wallis test, so that it can find no differences among the three samples. Here is an example (using R):

# generate & display fake data
set.seed(1234)  # for reproducibility
x1 = runif(5, 1, 4);  x2 = rnorm(5, 1, 1);  x3 = rnorm(5, 1, 5)
x123 = c(x1, x2, x3);  g123 = rep(1:3, each=5)
x12 = c(x1, x2);  g12 = rep(1:2, each=5)
stripchart(x123 ~ g123, pch="|", ylim=c(.5,3.5))


The two-sample Mann-Whitney-Wilcoxon test and the Kruskal-Wallis test both find a significant difference between x1 and x2at the 5% level, with P-values about 3%:

wilcox.test(x12 ~ g12)

Wilcoxon rank sum test

data:  x12 by g12
W = 23, p-value = 0.03175
alternative hypothesis: true location shift is not equal to 0

kruskal.test(x12 ~ g12)

Kruskal-Wallis rank sum test

data:  x12 by g12
Kruskal-Wallis chi-squared = 4.8109, df = 1, p-value = 0.02828


However, the Kruskal-Wallis test does not find any significant differences among x1, x2, and x3 at the 5% level:

kruskal.test(x123 ~ g123)

Kruskal-Wallis rank sum test

data:  x123 by g123
Kruskal-Wallis chi-squared = 5.58, df = 2, p-value = 0.06142

• the situation has nothing to do with violating assumptions of K-W test (it has none, unless one wants to use it as a test of medians, which it is not), nor with introducing variability (actually the 3rd sample hides the variability which made the K-W test with the first 2 samples significant). It has to do with the fact that the 3rd sample introduces a non-transitive relqationship between the p-values, and that results in the K-W test losing power. ... Commented Apr 24 at 21:25
• If I call p12 the p-value of 1 against 2 (p=.028), and p23 the p-value of 2 against 3 (p=.12), the p-value of 1 against 3 is also p13=.12 (all using 2-sample K-W tests). That is non-transitive: that 3rd p-value should be < the smallest of p12 and p23, and it is not... That make the K-W test misbehave... Commented Apr 24 at 21:26

You may want to look at this post on CV K-W as a test of stochastic superiority?
It reports a similar situation; start with 2 groups which are significantly different with a 2-sample K-W test. Add a 3rd group, and then the K-W test becomes non-significant... The reason is the non-transitivity of the stochastic superiority property: One can have A>B, and B>C, but C>A (read ">" as "is stochastically superior", meaning $$P(X_A)>P(X_B)>.5$$ (a random sample from A will be greater than a random sample from B more than 50% of the time). Yes, a rock/paper/scissors situation.
When such a non-transitive set of samples is fed to the K-W test, it loses statistical power (sometimes to an extreme point). You can also look at this paper "Kruskal–Wallis, Multiple Comparisons and Efron Dice", Bruce M. Brown and Thomas P. Hettmansperger https://api.semanticscholar.org/CorpusID:55698326 which explains this very well... This can be understood intuitively by remarking that the K-W test runs a sort of F test (ratio of quadratic terms based on ranks). But the denominator lumps all the samples into 1 (residuals, so to speak), and loses all information about what belong to which group; so with non-transitive superiorities, that lumping loses the difference.

The example in the previous answer actually demonstrates this. Sample is significantly superior to sample 2 (1 "beats" 2 23 out of 25 times). 2 is also superior to 3, but not at the 0.05 significance level (2 beats 3 only 20 out of 25 times, so p-value is only .1). For strict transitivity, one would expect 1 to "beat 3" even more so than 1 "beat" 2. But 1 "beats" 3 only 20 times out of 25. It is superior, but not anymore significantly so. That 3rd sample is not strictly transitivite. Hence the K-W test loses power... Hence, the difference which was apparent with only the first 2 samples, disappears with the 3rd...

Bottom line, K-W works only with strictly transitive relationships between the groups. Otherwise, it loses power.

Note that this has nothing to do with violating assumptions of K-W (there are none, unless you want to interpret it as a test on medians, which it is not), or introducing extra variability (which it does not; in fact the 3rd group masks the variability which existed with 2 groups).