Which result to choose when Kruskal-Wallis and Mann-Whitney seem to return contradicting results?

Question

I have these groups where the values are responses to a 10-point Likert item:

g1 <- c(10,9,10,9,10,8,9)
g2 <- c(4,9,4,9,8,8,8)
g3 <- c(9,7,9,4,8,9,10)

Therefore I used Kruskal-Wallis to determine any differences between responses in the groups, and the result was:

Kruskal-Wallis chi-squared = 5.9554, df = 2, p-value = 0.05091

However, if I run an exact Mann-Whitney test between groups g1 and g2 I get:

Exact Wilcoxon Mann-Whitney Rank Sum Test (using coin::wilcox_test)
Z = 2.3939, p-value = 0.02797

which returns a significant difference at alpha = 0.05.

Which test should I choose, and why?

Answer 1

I agree with Michael Chernick's answer, but think that it can be made a little stronger. Ignore the 0.05 cutoff in most circumstances. It is only relevant to the Neyman-Pearson approach which is largely irrelevant to the inferential use of statistics in many areas of science.

Both tests indicate that your data contains moderate evidence against the null hypothesis. Consider that evidence in light of whatever you know about the system and the consequences that follow from decisions (or indecision) about the state of the real world. Argue a reasoned case and proceed in a manner that acknowledges the possibility of subsequent re-evaluation.

I explain more in this paper: http://www.ncbi.nlm.nih.gov/pubmed/22394284

[Addendum added Nov 2019: I have a new reference that explains the issues in more detail https://arxiv.org/abs/1910.02042v1 ]

Answer 2

The Mann-Whitney or Wilcoxon test compares two groups while the Kruskal-Wallis test compares 3. Just like in the ordinary ANOVA with three or more groups the procedure generally suggested is to do the overall ANOVA F test first and then look at pairwise comparisons in case there is a significant difference. I would do the same here with the nonparametric ANOVA. My interpetation of your result is that there is marginally a significant difference between groups at level 0.05 and if you accept that then the difference based on the Mann-Whitney test indicates that it could be attributed to g$_1$ and g$_2$ being significantly different.

Don't get hung up with the magic of the 0.05 significance level! Just because the Kruskal-Wallis test gives p-value slightly over 0.05, don't take that to mean that there is no statistically significant difference between the groups. Also the fact that the Mann-Whitney test gives a p-value for the difference between g$_1$ and g$_2$ a little below 0.03 does not somehow make the difference between the two groups highly significant. Both p-values are close to 0.05. A slightly different data set could easily change to Kruskal-Wallis p-value by that much.

Any thought you might have that the results are contradictory would have to come from thinking of a 0.05 cut off as black and white boundary with no gray area in the neighborhood of 0.05. I think these results are reasonable and quite compatible.

Answer 3

Results of Kruskal-Wallis and Mann-Whitney U test may differ because

• The ranks used for the Mann-Whitney U test are not the ranks used by the Kruskal-Wallis test; and
• The rank sum tests do not use the pooled variance implied by the Kruskal-Wallis null hypothesis.

Hence, it is not recommended to use Mann-whitney U test as a post hoc test after Kruskal-Wallis test.

Other tests like Dunn's test (commonly used), Conover-Iman and Dwass-Steel-Citchlow-Fligner tests cane be used as post-hoc test for kruskal-wallis test.

Answer 4

This is in answer to @vinesh as well as looking at the general principle in the original question.

There are really 2 issues here with multiple comparisons: as we increase the number of comparisons being made we have more information which makes it easier to see real differences, but the increased number of comparisons also makes it easier to see differences that don't exist (false positives, data dredging, torturing the data until it confesses).

Think of a class with 100 students, each of the students is given a fair coin and told to flip the coin 10 times and use the results to test the null hypothesis that the proportion of heads is 50%. We would expect p-values to range between 0 and 1 and just by chance we would expect to see around 5 of the students get p-values less than 0.05. In fact we would be very surprised if none of them obtained a p-value less than 0.05 (less than 1% chance of that happening). If we only look at the few significant values and ignore all the others then we will falsely conclude that the coins are biased, but if we use a technique that takes into account the multiple comparisons then we will likely still judge correctly that the coins are fair (or at least fail to reject that they or fair).

On the other hand, consider a similar case where we have 10 students rolling a die and determining if the value is in the set {1,2,3} or the set {4,5,6} each of which will have 50% chance each roll if the die is fair (but could be different if the die is rigged). All 10 students compute p-values (null is 50%) and get values between 0.06 and 0.25. Now in this case none of them reached the magic 5% cut-off, so looking at any individual students results will not result in a non-fair declaration, but all the p-values are less than 0.5, if all the dice are fair then the p-values should be uniformly distributed and have a 50% chance of being above 0.5. The chance of getting 10 independent p-values all less than 0.5 when the nulls are true is less that the magic 0.05 and this suggests that the dice are biased, we just did not have enough power to detect this in the individual trials, but grouping the information shows the null is false.

Now coin flipping and die rolling are a bit contrived, so a different example: I have a new drug that I want to test. My budget allows me to test the drug on 1,000 subjects (this will be a paired comparison with each subject being their own control). I am considering 2 different study designs, in the first I recruite 1,000 subjects do the study and report a single p-value. In the second design I recruite 1,000 subjects but break them into 100 groups of 10 each, I do the study on each of the 100 groups of 10 and compute a p-value for each group (100 total p-values). Think about the potential differences between the 2 methodologies and how the conclusions could differ. An objective approach would require that both study designs lead to the same conclusion (given the same 1,000 patients and everything else is the same).

@mljrg, why did you choose to compare g1 and g2? If this was a question of interest before collecting any data then the MW p-value is reasonable and meaningful, however if you did the KW test, then looked to see which 2 groups were the most different and did the MW test only on those that looked the most different, then the assumptions for the MW test were violated and the MW p-value is meaningless and the KW p-value is the only one with potential meaning.