I have these groups where the values are responses to a 10-point Likert item:

g1 <- c(10,9,10,9,10,8,9)
g2 <- c(4,9,4,9,8,8,8)
g3 <- c(9,7,9,4,8,9,10)

Therefore I used Kruskal-Wallis to determine any differences between responses in the groups, and the result was:

Kruskal-Wallis chi-squared = 5.9554, df = 2, p-value = 0.05091

However, if I run an exact Mann-Whitney test between groups g1 and g2 I get:

Exact Wilcoxon Mann-Whitney Rank Sum Test (using coin::wilcox_test)
Z = 2.3939, p-value = 0.02797

which returns a significant difference at alpha = 0.05.

Which test should I choose, and why?


I agree with Michael Chernick's answer, but think that it can be made a little stronger. Ignore the 0.05 cutoff in most circumstances. It is only relevant to the Neyman-Pearson approach which is largely irrelevant to the inferential use of statistics in many areas of science.

Both tests indicate that your data contains moderate evidence against the null hypothesis. Consider that evidence in light of whatever you know about the system and the consequences that follow from decisions (or indecision) about the state of the real world. Argue a reasoned case and proceed in a manner that acknowledges the possibility of subsequent re-evaluation.

I explain more in this paper: http://www.ncbi.nlm.nih.gov/pubmed/22394284

[Addendum added Nov 2019: I have a new reference that explains the issues in more detail https://arxiv.org/abs/1910.02042v1 ]

| cite | improve this answer | |
  • $\begingroup$ @MichaelChernick I have became to learn from you that there is much more on statistics than just looking for "p<0.05". Michael Lew: I've downloaded your paper and will give it a read for sure. I'll follow your suggestion to have a good reasoning about my data in this situation. Thank you all! $\endgroup$ – mljrg Aug 10 '12 at 0:51
  • 3
    $\begingroup$ @MichaelLew I don't share your dim view of the Neyman-Pearson approach to hypothesis testing. I still think it is fundamental to frequentist inference. It is only the strict adherence to the 0.05 level that I object to. $\endgroup$ – Michael R. Chernick Aug 10 '12 at 1:15
  • $\begingroup$ @MichaelChernick So, are you saying that one should choose a cutoff for significance prior to the experiment, or that you can choose it after the results are in. The first is OK, but the second is not. Neyman-Pearson approach deals with error rates, and the type I error rate is only protected when the cutoff for significance is chosen in advance. Thus if you advise someone that a little over 0.05 is close enough because they might have chosen a higher cutoff, then you are not actually using the Neyman-Pearson approach, but an ill-formed hybrid approach as I explain in the paper linked. $\endgroup$ – Michael Lew Aug 11 '12 at 0:29
  • $\begingroup$ People can choose 0.01, 0.05 or 0.10 if they want. This should be done without being influenced by the data. But the choice of 0.01 or 0.05 is not the issue I refer to. It is the black and white belief in the significance level as those 0.049 means statistical significance and 0.0501 is not! $\endgroup$ – Michael R. Chernick Aug 11 '12 at 0:44
  • $\begingroup$ Scientists are interesting in evidence but they are not hung up on the methodology used to decide significance. $\endgroup$ – Michael R. Chernick Aug 11 '12 at 0:45

The Mann-Whitney or Wilcoxon test compares two groups while the Kruskal-Wallis test compares 3. Just like in the ordinary ANOVA with three or more groups the procedure generally suggested is to do the overall ANOVA F test first and then look at pairwise comparisons in case there is a significant difference. I would do the same here with the nonparametric ANOVA. My interpetation of your result is that there is marginally a significant difference between groups at level 0.05 and if you accept that then the difference based on the Mann-Whitney test indicates that it could be attributed to g$_1$ and g$_2$ being significantly different.

Don't get hung up with the magic of the 0.05 significance level! Just because the Kruskal-Wallis test gives p-value slightly over 0.05, don't take that to mean that there is no statistically significant difference between the groups. Also the fact that the Mann-Whitney test gives a p-value for the difference between g$_1$ and g$_2$ a little below 0.03 does not somehow make the difference between the two groups highly significant. Both p-values are close to 0.05. A slightly different data set could easily change to Kruskal-Wallis p-value by that much.

Any thought you might have that the results are contradictory would have to come from thinking of a 0.05 cut off as black and white boundary with no gray area in the neighborhood of 0.05. I think these results are reasonable and quite compatible.

| cite | improve this answer | |
  • 2
    $\begingroup$ You will better communicate your answer when you re-read it for errors (in punctuation, grammar, typography, and spelling) and use effective formatting. Please review the Markdown help page. $\endgroup$ – whuber Aug 9 '12 at 21:44
  • $\begingroup$ The more classic view is that you failed to find statistical significance with your first test, so you should not report (in a professional publication) any further tests as statistically significant indications of between group differences. To do so is to use an alpha other than .05. This is particularly problematic (from the classical view) because you did not choose the higher alpha before conducting the test, so your alpha is unknown. Of course, when you try to understand your data, to guide your own future research program, you can take note of the difference between groups 1 and 2. $\endgroup$ – Joel W. Aug 9 '12 at 21:53
  • $\begingroup$ @JoelW. Are you trying to tell me that 0.05091 is really different from 0.05? Anyway my point is not how to report the conclusions but rather to say that the two tests don't conflict. I agree that how you analyze the data should be specified in advance before looking at the data. $\endgroup$ – Michael R. Chernick Aug 9 '12 at 22:11
  • 1
    $\begingroup$ @whuber Sorry for not editing the post earlier. I hope it looks a lot better now. $\endgroup$ – Michael R. Chernick Aug 9 '12 at 22:18
  • $\begingroup$ @JoelW Your 'more classic' view is actually Neyman's 'inductive behaviour' approach to inference. It is relevant to a small subset of the uses of statistics in support of inference. It is most unfortuate that it is presented so often as being classic. $\endgroup$ – Michael Lew Aug 9 '12 at 22:51

Results of Kruskal-Wallis and Mann-Whitney U test may differ because

  • The ranks used for the Mann-Whitney U test are not the ranks used by the Kruskal-Wallis test; and
  • The rank sum tests do not use the pooled variance implied by the Kruskal-Wallis null hypothesis.

Hence, it is not recommended to use Mann-whitney U test as a post hoc test after Kruskal-Wallis test.

Other tests like Dunn's test (commonly used), Conover-Iman and Dwass-Steel-Citchlow-Fligner tests cane be used as post-hoc test for kruskal-wallis test.

| cite | improve this answer | |

This is in answer to @vinesh as well as looking at the general principle in the original question.

There are really 2 issues here with multiple comparisons: as we increase the number of comparisons being made we have more information which makes it easier to see real differences, but the increased number of comparisons also makes it easier to see differences that don't exist (false positives, data dredging, torturing the data until it confesses).

Think of a class with 100 students, each of the students is given a fair coin and told to flip the coin 10 times and use the results to test the null hypothesis that the proportion of heads is 50%. We would expect p-values to range between 0 and 1 and just by chance we would expect to see around 5 of the students get p-values less than 0.05. In fact we would be very surprised if none of them obtained a p-value less than 0.05 (less than 1% chance of that happening). If we only look at the few significant values and ignore all the others then we will falsely conclude that the coins are biased, but if we use a technique that takes into account the multiple comparisons then we will likely still judge correctly that the coins are fair (or at least fail to reject that they or fair).

On the other hand, consider a similar case where we have 10 students rolling a die and determining if the value is in the set {1,2,3} or the set {4,5,6} each of which will have 50% chance each roll if the die is fair (but could be different if the die is rigged). All 10 students compute p-values (null is 50%) and get values between 0.06 and 0.25. Now in this case none of them reached the magic 5% cut-off, so looking at any individual students results will not result in a non-fair declaration, but all the p-values are less than 0.5, if all the dice are fair then the p-values should be uniformly distributed and have a 50% chance of being above 0.5. The chance of getting 10 independent p-values all less than 0.5 when the nulls are true is less that the magic 0.05 and this suggests that the dice are biased, we just did not have enough power to detect this in the individual trials, but grouping the information shows the null is false.

Now coin flipping and die rolling are a bit contrived, so a different example: I have a new drug that I want to test. My budget allows me to test the drug on 1,000 subjects (this will be a paired comparison with each subject being their own control). I am considering 2 different study designs, in the first I recruite 1,000 subjects do the study and report a single p-value. In the second design I recruite 1,000 subjects but break them into 100 groups of 10 each, I do the study on each of the 100 groups of 10 and compute a p-value for each group (100 total p-values). Think about the potential differences between the 2 methodologies and how the conclusions could differ. An objective approach would require that both study designs lead to the same conclusion (given the same 1,000 patients and everything else is the same).

@mljrg, why did you choose to compare g1 and g2? If this was a question of interest before collecting any data then the MW p-value is reasonable and meaningful, however if you did the KW test, then looked to see which 2 groups were the most different and did the MW test only on those that looked the most different, then the assumptions for the MW test were violated and the MW p-value is meaningless and the KW p-value is the only one with potential meaning.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.