# Is Kruskal-Wallis really a good test of stochastic superiority?

After much reading/googling, I reached the conclusion that the Kruskal-Wallis test (K-W) is a test of stochastic superiority (or, equivalently, of stochastic equivalence). Rejecting the null implies that at least 1 population is stochastically superior to at least another 1 (E.g. from Wikipedia, "A significant Kruskal–Wallis test indicates that at least one sample stochastically dominates one other sample".)

One interesting property of stochastic superiority is the fact that this is not a transitive property.

So, for fun (?), I created 4 samples from Efron's dice (non-transitive!). I replicated the values 6 times, for a sample size of 48 each. To avoid so many ties, I added to each a small amount of gaussian noise ($$N(0,.1)$$).This gave me the 4 samples below

I then ran a Kruskal-Wallis test on these 4 samples. The p-value was 0.974. Certainly not significant... I also ran the Dunn MCT, which said that there were no significant group differences (of course, given the p-value).

I then ran all 6 Mann-Whitney U tests. I used Sidak's MCC, which gave me an adjusted $$\alpha'=0.0085$$ for 6 comparisons (0.00833 for Bonferroni). Sample A was stochastically superior to B, which was superior to C, which was superior to D, which was superior to A (as we would expect from Efron's dice). The other 2 (A-C and B-D) showed no significance. For all the significant M-W U tests, the p-value was $$0.005 < 0.0085$$).

This leaves me with a non-significant K-W, even though there are 4 significant M-W U. Which do you believe? (I know I believe the M-W U tests, fwiw)

I know that, hand-wavingly speaking, the K-W test's statistic is akin to an F test, but on ranks. But by "blurring" the groups together, does it not lose the specific shape differences of the various samples, which made them stochastically different?

Does that mean that K-W is NOT a good omnibus test of stochastic superiority? Then what does it test?

## Updated 4/23/2024

I ran a few more tests.

• First a K-W with just the first 2 samples (A & B). p=0.005 (Just like M-W U, as expected.
• Then a K-W with the first 3 samples (A, B & C). p=0.456 (?) So K-W told me that there was a difference between A & B (note: I do not specify what the difference is, on purpose: stochastic superiority, median, distribution, ???. But there was a difference between A & B). However, when I add a 3rd sample, this significant difference disappears?
• Thinking that maybe K-W loses power when more groups are added (??), I quadrupled the sample size to 192 per group. Re-ran K-W on these 4, and got p=0.828. The M-W U between A & B (e.g.) gave p=0.000000016
• Then, trying to see if the inability of K-W to find significance was because of the non-transitivity of stochastic superiority, I kept samples A & B at a size of 198, but dropped C&D (which contain non-transitive values) at 48. This new K-W test gave a p-value of 0.007. Finally!

What is happening with K-W? And why?

## Updated 4/24/2024

After digging around a bit more, I came around this very informative paper "Kruskal–Wallis, Multiple Comparisons and Efron Dice", Bruce M. Brown and Thomas P. Hettmansperger https://api.semanticscholar.org/CorpusID:55698326 which explains the situation very well.

The issue has to do with the non-transitive property of stochastic superiority. When there are such non-transitive situations (the authors call these "circularities", but to me that gives the wrong image, so I will stick with the cumbersome "non-transitivities"), the power of the K-W test diminishes, sometimes dramatically, like in the example of my question.

I doubt that "non-transitivities" need to be of the extreme "rock/paper/scissors" type as I used in the original question. I think they can be much more subtle, as in the example below: we have 3 samples A,B & C (5 samples each).[![K-W counter example][2]][2]

Because the sample size is (very) small, one can directly look at all the pair-wise comparisons. A "beats" B 23 out of 25 times, and B "beats" C 20 out of 25 times. One would expect A to beat C handily? But this occurs (only) 20 out of 25 times. A strict transitivity property has not been respected. The 2-sample K-W test (A,B) was significant (p=0.028), but the 3-sample K-W test (A,B,C) gives p=0.061.

• What type of stochastic dominance are you interested in? Apr 23 at 16:33
• The same one used for M-W U, or the original Efron's dice, namely $P(X_A>X_B)>.5$. A randomly selected value from population A will be greater than a randomly selected value from population B more than 50% of the time. Apr 23 at 16:46
• That's not stochastic dominance, which is $\forall x, F_A(x) \leq F_B(x)$ with strict inequality for at least one $x$. Apr 24 at 2:07
• That is exactly why I called it stochastic superiority! (and not dominance) @Galen used the term "dominance". Wikipedia also used the term, but I did not... And K-W test does not demonstrate dominance (of the first order, which is the definition you gave), it demonstrates superiority, according to the definition I gave... It is trivial to create counter-examples of significant K-W (or MW-U) tests, where $F_A(x)$ is not $<= F_B(x)$ for all x's. Apr 24 at 19:47
• That's fair enough, @jginestet. Thank you for defining the term "stochastic superiority". Apr 24 at 19:49

I don't like to spend much time with special cases, and for the multi-group problem I use the generalization of the K-W test, the proportional odds ordinal logistic semiparametric model. The model also exposes what is going on with regard to transitivity, in a way. To get the K-W test (actually a better version of it that handles ties better and allows for covariate adjustment) one puts $$k-1$$ indicator variables in the model for $$k$$ groups. An example is here. When you translate the resulting odds ratios to concordance probabilities the transitivity problem persists. But when viewing effects on the scale of logit of exceedance probabilities, i.e., $$\text{logit}(\Pr(Y \geq y))$$ as the proportional odds model is originally stated, you'll see that the group effects on this scale are transitive.