3
$\begingroup$

I have done this analysis where in Figure A-D. the Kruskal-Wallis global p-value is non-significant, but the p-value for pair CS and SCS in Figure A however is significant (at P<0.05). I read somewhere that if the global p-value is non-significant we don't report post-hoc pairwise tests. So In figure A, B, C, D and H, should I just delete the p-values for paired tests as the global p-values in these plots are not significant? Could someone please clarify this?

enter image description here

$\endgroup$
12
  • 1
    $\begingroup$ You also do not say if you perform any adjustments for multiple comparisons? (My packages dunn.test and conover.test implement this functionality in truly post hoc tests for K-W.) $\endgroup$
    – Alexis
    Oct 25 '19 at 21:45
  • 1
    $\begingroup$ Okay. At least to me that webpage is unclear as to what test it's performing in this case. It might be performing pairwise Wilcoxon-Mann-Whitney tests. This is probably less advantageous than the Dunn test (1964) or some other tests, and that may play into the test finding significant difference between groups. $\endgroup$ Oct 25 '19 at 21:46
  • 1
    $\begingroup$ Yeah, simple pairwise wilcox.tests don't reflect a post-hoc environment for the Kruskal-Wallis test as well as e.g. the Dunn test (1964). It could be part of the disconnect in the results. $\endgroup$ Oct 25 '19 at 22:23
  • 1
    $\begingroup$ Specifically, pairwise rank sum tests do not (1) use the same rankings of the data used by the Kruskal-Wallis test, and (2) do not use the pooled variance implied by the null hypothesis of the Kruskal-Wallis test. $\endgroup$
    – Alexis
    Oct 25 '19 at 22:25
  • 1
    $\begingroup$ I like Dunn test (1964). It makes sense to me, but I'm not an expert. $\endgroup$ Oct 25 '19 at 22:38
5
$\begingroup$

In general, if you use an omnibus test, such as an ANOVA F-test or a Kruskal-Wallis H-test, it is illogical and poor practice to conduct pairwise comparisons when you fail to reject the null hypothesis on the omnibus test. Conducting the comparisons flies in the face of the omnibus: insufficient evidence to conclude differences does not warrant further investigation, as a general rule.

Usually, I would say report analyses you run, but in this case (which is different from selective reporting), the post-hoc p-values are inappropriate to interpret and should be omitted. The omnibus p-value is appropriate since this is the “gatekeeper” test.

$\endgroup$
3
  • $\begingroup$ With the caveat that many tests we think of as post-hoc could also be used as the primary analysis, without first conducting an omnibus test. A discussion relating to traditional anova can be found at the following link. I've seen this discussed with nonparametric tests as well, but I don't have a reference or link. stats.stackexchange.com/questions/9751/… $\endgroup$ Oct 26 '19 at 13:11
  • $\begingroup$ Right, I understand this. My point is if someone uses an omnibus, it is presumably to help mitigate multiplicity and act as a gatekeeper. That was done in this case, so getting tied up in an apparent paradox isn't particularly helpful. I say "apparent paradox" because the tests ask different questions and so "conflicting results" aren't really conflicting because different questions are asked. The question posed here was whether pairwise comparisons are appropriate (presumably without preplanning) in light of a nonsignificant omnibus test; the short answer is no. 1/2 $\endgroup$
    – LSC
    Oct 26 '19 at 13:44
  • $\begingroup$ The logical flow of testing helps prevent p-hacking or "just because" "inferences" from occurring. This is also why in my response I said, "in general", because this is a strongly generalized answer but it applies specifically to the OP's scenario. If the OP had given a scenario of pre-planned contrasts irrespective of the omnibus, then a different answer would be helpful. 2/2 $\endgroup$
    – LSC
    Oct 26 '19 at 13:47
3
$\begingroup$

Illustrating @LSC's answer (+1), here is an example to show that doing ad hoc two-sample Wilcoxon tests can lead to 'false discoveries' if a Kruskal-Wallis test is not significant.

Suppose we have five groups of size $n = 20$ with values distributed uniformly on the same interval. So, ideally we should not find any significant differences at all.

Let the significance level of all tests be 10%. Then the K-W test will reject $H_0$ that all five groups come from the same population distribution about 10% of the time.

If we were to do ad hoc two sample Wilcoxon tests to compare the two groups out of five that have the most different medians, we would reject the null hypothesis that those two groups differ another 10% of the time.

A simulation in R of 10,000 such datasets, each with five groups of 20 observations, is shown below:

set.seed(2019)
m = 10^4;  p.kw = p.wx = numeric(m)
for(i in 1:m) {
  x1 = runif(20, 10, 25);  x2 = runif(20, 10, 25)
  x3 = runif(20, 10, 25);  x4 = runif(20, 10, 25)
  x5 = runif(20, 10, 25)
  p.kw[i] = kruskal.test(list(x1,x2,x3,x4,x5))$p.val
  MAT=rbind(x1,x2,x3,x4,x5)
  h = apply(MAT,1,median)
  mx = which(h==max(h));  mn = which(h==min(h))
  p.wx[i] = wilcox.test(MAT[mx,1],MAT[mn,])$p.val
}
mean(p.kw<.1);  mean(p.wx<.1)
[1] 0.0947      # rejection rate for K-W
[1] 0.1017      # rej rate: all Wilcoxon for two extremes
mean(p.wx[p.kw > .1] < .1)
[1] 0.09632166  # rej rate: improper ad hoc Wilcoxon tests

It happens that one of the unwarranted ad hoc Wilcoxon rank sum tests occurred for the last of the 10,000 iterations. Boxplots for the five groups are shown, followed by a non-significant (10% level) K-W test and a significant two-sample Wilcoxon test comparins groups 2 and 3.

enter image description here

kruskal.test(list(x1,x2,x3,x4,x5))$p.value
[1] 0.3357694             # not significant at 10%
wilcox.test(x2,x3)$p.val
[1] 0.09108615            # 'significant' improper ad hoc test

Added per Comment: The P-value for a K-W test comparing the same two groups gives very nearly the same P-value as the Wilcoxon test above; both P-values essentially 0.9.

kruskal.test(list(x2,x3))$p.val
[1] 0.08835202
$\endgroup$
4
  • $\begingroup$ Thank you. Just wanted to be clear- I followed the tests illustrated here sthda.com/english/articles/24-ggpubr-publication-ready-plots/…. I used stat_compare_means(comparisons = my_comparisons, method = "kruskal.method"). I thought I used kruska.test for post-hoc comparisons also. Am I missing something here? $\endgroup$
    – MAPK
    Oct 25 '19 at 21:25
  • 1
    $\begingroup$ Non familiar with that exact procedure. Usually two-sample Wilcoxon tests ad hoc comparisons among levels after a significant K-W test (often with Bonferroni levels to protect against false discovery). However, K-W tests work for only 2 groups and then are very nearly the same as 2-sample Wilcoxon, so it might not make a difference which is used. $\endgroup$
    – BruceET
    Oct 25 '19 at 21:31
  • 1
    $\begingroup$ See K-W test for last example in my Answer, just added. $\endgroup$
    – BruceET
    Oct 25 '19 at 21:43
  • 1
    $\begingroup$ Thank you so much for this great answer. $\endgroup$
    – MAPK
    Oct 25 '19 at 21:47
0
$\begingroup$

There is a question underlying OP's example that is not a trivial one: If a test like KW assesses a hypothesis that all groups are (stochastically) equal, and if we find a non-significant result for KW, but then a significant result for pairwise tests, How is it that these tests are valid? Why would we ever want to use an omnibus test instead of jumping to pairwise tests?

Some of the comments on the original post partially address this situation.

One issue is that KW looks at the data in all groups simultaneously, whereas presumably the software in the OP's example is using pairwise Mann-Whitney tests which look at two groups at a time while ignoring the other groups. So, simply, they are addressing the analysis in different ways. Usually it is considered better to look at all groups simultaneously. This can also avoid potential incommensurate pairwise results (where e.g. A > B, B > C, and C > A). As mentioned in the comments on the original post, we might use Dunn's test (1964) or Conover-Iman instead of a simple pairwise approach.

Likely another issue here is that the post-hoc approach used may not have adjusted p values for multiple tests. If this is the case, then the likelihood of finding a significant result in the pairwise tests, just by chance, is elevated.

I also think there's some sense to the idea that if are really interested in comparing groups, that we can skip the omnibus test and look at group comparisons (Link1, Link2). In the context of KW, this logic may also apply to effect size statistics. That is, using pairwise or maximum Vargha and Delaney's A rather than epsilon-squared.

$\endgroup$
1
  • $\begingroup$ "How is it that these tests are valid? " I think this question skirts totally past the point that the tests may be entirely valid because they're asking different questions, so the "contradiction'' isn't real. You only perceive contradiction because you ran the different tests and fail to understand the questions that are formulated into the tests. $\endgroup$
    – LSC
    Oct 8 '20 at 0:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.