4
$\begingroup$

I have a non-normalized dependent variable and an independent variable broken down into 4 groups. As such, I used the Kruskal–Wallis analysis to look for significant differences in the ranks of the groups. The data look like the following:

\begin{array}{clc}\rm Group&\rm Size&\rm Means\\\hline 0&n=24&31.79\\ 1&n=13&26.65\\ 2&n=8 &15.94\\ 3&n=10&30.30\\ \rm Total& N=55\end{array}

I get an asymptotic significance of .103.

However, if I run a Mann–Whitney U on the same data, I see a clear significance between the 0 and 2 groups:

   0    24  18.81   451.50
   2    8   9.56    76.50
Total   32      

Mann-Whitney U  40.500
Wilcoxon W  76.500
Z   -2.416

Asymp. Sig. (2-tailed)  .016
Exact Sig. [2*(1-tailed Sig.)]  .013b

The only things I could come up with were that:

  • I may be running into some kind of confounding error with the Mann–Whitney U test, and thus the significance is a mistake, or that
  • The large difference in between the sample sizes is what is causing me to see significance with the Mann–Whitney, but not the K–W since it has a bit more "power" to tease out what is significant and what isn't.

Any help or guidance would be appreciated.

Improve this question
$\endgroup$
6
  • $\begingroup$ Is your table the means of the data (which is what "Means" suggests) or the means of the ranks of the data? There's several possible reasons why, but there's not really much to go on here that would allow us to identify which it might be. $\endgroup$
    – Glen_b
    Commented Jul 23, 2014 at 3:34
  • 1
    $\begingroup$ If you're trying to look at post-hoc for a K-W, take a look at Dunn's test $\endgroup$
    – Glen_b
    Commented Jul 23, 2014 at 3:54
  • $\begingroup$ @Glen_b this is the "Means Rank" of the data. @Glen/@Alexis: I understand the Dunn's test is the appropriate post hoc test for KW. But that is to be used when there is a significant difference seen in KW, correct? I did not see a difference when I did the KW. The significant difference was seen only when I did individual MWU's to compare the independent groups ("0" vs "2") $\endgroup$
    – user52641
    Commented Jul 23, 2014 at 13:19
  • 1
    $\begingroup$ I'm confused now - are you suggesting that by contrast with Dunn, it's somehow okay to use MW after KW doesn't reject? (If you seek something to do after KW doesn't reject, pairwise MW comparisons isn't it. What's the purpose of this comparison? If it's intellectual curiosity about the difference that's fine, but then my response about Dunn is on point (if you seek a pairwise comparison with consistency with KW, Dunn will come closer than MW). If it's an attempt to find something significant, you're data dredging.) $\endgroup$
    – Glen_b
    Commented Jul 23, 2014 at 18:14
  • 1
    $\begingroup$ Note that even with the most appropriate choice of pairwise comparison, you could get non-significance overall but significance on an individual pairwise test (were you to carry it out in spite of the non-rejection). This doesn't necessarily imply that the overall test "missed something" or that pairwise comparisons have "more power". $\endgroup$
    – Glen_b
    Commented Sep 25, 2015 at 1:44

2 Answers 2

Reset to default
9
$\begingroup$

Perhaps not surprising that you got disparate results since (ahem): the Mann-Whitney rank sum test is not an appropriate post hoc pairwise test for the Kruskal-Wallis omnibus test.

The rank sum test:

  1. Ignores the rankings used to conduct the Kruskal-Wallis test

  2. Does not account for the pooled variance implied by the null hypothesis of the Kruskal-Wallis test.

A popular and appropriate post hoc pairwise test is Dunn's test, which does employ the same rankings as the Kruskal-Wallis, and does account for pooled variance implied by the null hypothesis of the Kruskal-Wallis test. Another less well-known post hoc test is the Conover-Iman test, which is strictly more powerful that Dunn's test (assuming the Kruskal-Wallis test rejects its null hypothesis).

Dunn's test is implemented for Stata in the dunntest package (within Stata type net describe dunntest, from(http://www.alexisdinno.com/stata)), and for R in the dunn.test package. The Conover-Iman test is implemented within Stata in the conovertest package (within Stata type net describe conovertest, from(http://www.alexisdinno.com/stata)), and for R in the conover.test package.

Being concerned that you found disparate results between the Kruskal-Wallis and rank sum tests is a little like being concerned that you found disparate results between ANOVA on transformed data, and unpaired t tests without pooled variance on untransformed data; I think you are asking the wrong question.

Improve this answer
$\endgroup$
2
  • $\begingroup$ WouldMann-Whitney as a post hoc test be like doing post hoc t-tests after an ANOVA using the usual t.test(group1, group5) in R, rather than using the pooled variance estimate from all groups considered in the ANOVA? $\endgroup$
    – Dave
    Commented Jun 17, 2020 at 16:33
  • $\begingroup$ @Dave No, because the Mann-Whitney uses different rankings (i.e. different data) than the Kruskal-Wallis, and so is pretty divorced from the inferences of the KW test. Mann-Whitney is a pretty poor choice for a post hoc test to Kruskal-Wallis. The last paragraph of my answer really makes this point. $\endgroup$
    – Alexis
    Commented Jun 18, 2020 at 17:07
5
$\begingroup$

The Kruskal-Wallis test is the one that has a perfect multiplicity adjustment. Looking for pairwise differences when K-W is insignificant is dangerous. When you do want to look pairwise, heed the advice above to use a method that is consistent with K-W. A model-based approach that generalizes K-W is the proportional odds ordinal logistic model, which can be used to test any contrast you want plus give the global likelihood ratio $\chi^2$ test that is related to the K-W test.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for the answer Mr. Harrell. I understand what you mean. $\endgroup$
    – user52596
    Commented Jul 23, 2014 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.