8
$\begingroup$

After much reading/googling, I reached the conclusion that the Kruskal-Wallis test (K-W) is a test of stochastic superiority (or, equivalently, of stochastic equivalence). Rejecting the null implies that at least 1 population is stochastically superior to at least another 1 (E.g. from Wikipedia, "A significant Kruskal–Wallis test indicates that at least one sample stochastically dominates one other sample".)

One interesting property of stochastic superiority is the fact that this is not a transitive property.

So, for fun (?), I created 4 samples from Efron's dice (non-transitive!). I replicated the values 6 times, for a sample size of 48 each. To avoid so many ties, I added to each a small amount of gaussian noise ($N(0,.1)$).This gave me the 4 samples below Efron dice

I then ran a Kruskal-Wallis test on these 4 samples. The p-value was 0.974. Certainly not significant... I also ran the Dunn MCT, which said that there were no significant group differences (of course, given the p-value).

I then ran all 6 Mann-Whitney U tests. I used Sidak's MCC, which gave me an adjusted $\alpha'=0.0085$ for 6 comparisons (0.00833 for Bonferroni). Sample A was stochastically superior to B, which was superior to C, which was superior to D, which was superior to A (as we would expect from Efron's dice). The other 2 (A-C and B-D) showed no significance. For all the significant M-W U tests, the p-value was $0.005 < 0.0085$).

This leaves me with a non-significant K-W, even though there are 4 significant M-W U. Which do you believe? (I know I believe the M-W U tests, fwiw)

I know that, hand-wavingly speaking, the K-W test's statistic is akin to an F test, but on ranks. But by "blurring" the groups together, does it not lose the specific shape differences of the various samples, which made them stochastically different?

Does that mean that K-W is NOT a good omnibus test of stochastic superiority? Then what does it test?

Updated 4/23/2024

I ran a few more tests.

  • First a K-W with just the first 2 samples (A & B). p=0.005 (Just like M-W U, as expected.
  • Then a K-W with the first 3 samples (A, B & C). p=0.456 (?) So K-W told me that there was a difference between A & B (note: I do not specify what the difference is, on purpose: stochastic superiority, median, distribution, ???. But there was a difference between A & B). However, when I add a 3rd sample, this significant difference disappears?
  • Thinking that maybe K-W loses power when more groups are added (??), I quadrupled the sample size to 192 per group. Re-ran K-W on these 4, and got p=0.828. The M-W U between A & B (e.g.) gave p=0.000000016
  • Then, trying to see if the inability of K-W to find significance was because of the non-transitivity of stochastic superiority, I kept samples A & B at a size of 198, but dropped C&D (which contain non-transitive values) at 48. This new K-W test gave a p-value of 0.007. Finally!

What is happening with K-W? And why?

Updated 4/24/2024

After digging around a bit more, I came around this very informative paper "Kruskal–Wallis, Multiple Comparisons and Efron Dice", Bruce M. Brown and Thomas P. Hettmansperger https://api.semanticscholar.org/CorpusID:55698326 which explains the situation very well.

The issue has to do with the non-transitive property of stochastic superiority. When there are such non-transitive situations (the authors call these "circularities", but to me that gives the wrong image, so I will stick with the cumbersome "non-transitivities"), the power of the K-W test diminishes, sometimes dramatically, like in the example of my question.

I doubt that "non-transitivities" need to be of the extreme "rock/paper/scissors" type as I used in the original question. I think they can be much more subtle, as in the example below: we have 3 samples A,B & C (5 samples each).[![K-W counter example][2]][2]

Because the sample size is (very) small, one can directly look at all the pair-wise comparisons. A "beats" B 23 out of 25 times, and B "beats" C 20 out of 25 times. One would expect A to beat C handily? But this occurs (only) 20 out of 25 times. A strict transitivity property has not been respected. The 2-sample K-W test (A,B) was significant (p=0.028), but the 3-sample K-W test (A,B,C) gives p=0.061.

Improve this question
$\endgroup$
17
  • $\begingroup$ What type of stochastic dominance are you interested in? $\endgroup$
    – Galen
    Commented Apr 23 at 16:33
  • $\begingroup$ The same one used for M-W U, or the original Efron's dice, namely $P(X_A>X_B)>.5$. A randomly selected value from population A will be greater than a randomly selected value from population B more than 50% of the time. $\endgroup$
    – jginestet
    Commented Apr 23 at 16:46
  • 1
    $\begingroup$ That's not stochastic dominance, which is $\forall x, F_A(x) \leq F_B(x)$ with strict inequality for at least one $x$. $\endgroup$
    – jbowman
    Commented Apr 24 at 2:07
  • $\begingroup$ That is exactly why I called it stochastic superiority! (and not dominance) @Galen used the term "dominance". Wikipedia also used the term, but I did not... And K-W test does not demonstrate dominance (of the first order, which is the definition you gave), it demonstrates superiority, according to the definition I gave... It is trivial to create counter-examples of significant K-W (or MW-U) tests, where $F_A(x)$ is not $<= F_B(x)$ for all x's. $\endgroup$
    – jginestet
    Commented Apr 24 at 19:47
  • $\begingroup$ That's fair enough, @jginestet. Thank you for defining the term "stochastic superiority". $\endgroup$
    – Galen
    Commented Apr 24 at 19:49

1 Answer 1

Reset to default
2
$\begingroup$

I don't like to spend much time with special cases, and for the multi-group problem I use the generalization of the K-W test, the proportional odds ordinal logistic semiparametric model. The model also exposes what is going on with regard to transitivity, in a way. To get the K-W test (actually a better version of it that handles ties better and allows for covariate adjustment) one puts $k-1$ indicator variables in the model for $k$ groups. An example is here. When you translate the resulting odds ratios to concordance probabilities the transitivity problem persists. But when viewing effects on the scale of logit of exceedance probabilities, i.e., $\text{logit}(\Pr(Y \geq y))$ as the proportional odds model is originally stated, you'll see that the group effects on this scale are transitive.

Thus transitivity is a transformation-dependent idea.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.