3
$\begingroup$

We have two groups that are significantly different (tested with Mann-Whitney). When a third group is added, the Kruskal-Wallis tests provides a non-significant result.

This implies that two or more groups were not significantly different from each other; hence the non-significant result.

or

Does this imply that we do not have enough power to detect any effect (type-1 error) and the third group is likely not different from the other two groups? So the result is inconclusive.

$\endgroup$
3
  • 3
    $\begingroup$ Let's flag the elementary but fundamental principle: Plot the data to see what is going on. Further: why not show us the data? $\endgroup$
    – Nick Cox
    Commented Apr 6, 2019 at 8:15
  • $\begingroup$ There are multiple possible explanations; we can't guess which one(s) will be the case for you. (Most of the possible explanations are explored in answers to other questions already on site) $\endgroup$
    – Glen_b
    Commented Apr 6, 2019 at 8:29
  • $\begingroup$ You are using the incorrect post hoc test. Mann-Whitney (a) does not use the same rankings as the K-S, so no surprise you are getting strange results, and (b) does not use the pooled variance assumed by the K-S null. Try Dunn's test or, even better, the Conover-Iman test. $\endgroup$
    – Alexis
    Commented Apr 15, 2023 at 20:10

2 Answers 2

2
$\begingroup$

One possibility is that the third sample introduces extra variability or violates assumptions for the Kruskal-Wallis test, so that it can find no differences among the three samples. Here is an example (using R):

# generate & display fake data
set.seed(1234)  # for reproducibility 
x1 = runif(5, 1, 4);  x2 = rnorm(5, 1, 1);  x3 = rnorm(5, 1, 5)
x123 = c(x1, x2, x3);  g123 = rep(1:3, each=5)
x12 = c(x1, x2);  g12 = rep(1:2, each=5)
stripchart(x123 ~ g123, pch="|", ylim=c(.5,3.5))

enter image description here

The two-sample Mann-Whitney-Wilcoxon test and the Kruskal-Wallis test both find a significant difference between x1 and x2at the 5% level, with P-values about 3%:

wilcox.test(x12 ~ g12)

        Wilcoxon rank sum test

data:  x12 by g12
W = 23, p-value = 0.03175
alternative hypothesis: true location shift is not equal to 0


kruskal.test(x12 ~ g12)

        Kruskal-Wallis rank sum test

data:  x12 by g12
Kruskal-Wallis chi-squared = 4.8109, df = 1, p-value = 0.02828

However, the Kruskal-Wallis test does not find any significant differences among x1, x2, and x3 at the 5% level:

kruskal.test(x123 ~ g123)

        Kruskal-Wallis rank sum test

data:  x123 by g123
Kruskal-Wallis chi-squared = 5.58, df = 2, p-value = 0.06142
$\endgroup$
2
  • $\begingroup$ the situation has nothing to do with violating assumptions of K-W test (it has none, unless one wants to use it as a test of medians, which it is not), nor with introducing variability (actually the 3rd sample hides the variability which made the K-W test with the first 2 samples significant). It has to do with the fact that the 3rd sample introduces a non-transitive relqationship between the p-values, and that results in the K-W test losing power. ... $\endgroup$
    – jginestet
    Commented Apr 24 at 21:25
  • $\begingroup$ If I call p12 the p-value of 1 against 2 (p=.028), and p23 the p-value of 2 against 3 (p=.12), the p-value of 1 against 3 is also p13=.12 (all using 2-sample K-W tests). That is non-transitive: that 3rd p-value should be < the smallest of p12 and p23, and it is not... That make the K-W test misbehave... $\endgroup$
    – jginestet
    Commented Apr 24 at 21:26
2
$\begingroup$

You may want to look at this post on CV K-W as a test of stochastic superiority?
It reports a similar situation; start with 2 groups which are significantly different with a 2-sample K-W test. Add a 3rd group, and then the K-W test becomes non-significant... The reason is the non-transitivity of the stochastic superiority property: One can have A>B, and B>C, but C>A (read ">" as "is stochastically superior", meaning $P(X_A)>P(X_B)>.5$ (a random sample from A will be greater than a random sample from B more than 50% of the time). Yes, a rock/paper/scissors situation.
When such a non-transitive set of samples is fed to the K-W test, it loses statistical power (sometimes to an extreme point). You can also look at this paper "Kruskal–Wallis, Multiple Comparisons and Efron Dice", Bruce M. Brown and Thomas P. Hettmansperger https://api.semanticscholar.org/CorpusID:55698326 which explains this very well... This can be understood intuitively by remarking that the K-W test runs a sort of F test (ratio of quadratic terms based on ranks). But the denominator lumps all the samples into 1 (residuals, so to speak), and loses all information about what belong to which group; so with non-transitive superiorities, that lumping loses the difference.

The example in the previous answer actually demonstrates this. Sample is significantly superior to sample 2 (1 "beats" 2 23 out of 25 times). 2 is also superior to 3, but not at the 0.05 significance level (2 beats 3 only 20 out of 25 times, so p-value is only .1). For strict transitivity, one would expect 1 to "beat 3" even more so than 1 "beat" 2. But 1 "beats" 3 only 20 times out of 25. It is superior, but not anymore significantly so. That 3rd sample is not strictly transitivite. Hence the K-W test loses power... Hence, the difference which was apparent with only the first 2 samples, disappears with the 3rd...

Bottom line, K-W works only with strictly transitive relationships between the groups. Otherwise, it loses power.

Note that this has nothing to do with violating assumptions of K-W (there are none, unless you want to interpret it as a test on medians, which it is not), or introducing extra variability (which it does not; in fact the 3rd group masks the variability which existed with 2 groups).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.