0
$\begingroup$

I performed Krusal-Wallis test in SPSS and got a borderline significant P (0.042). then, the post hoc pairwise comparison with Bonferroni-Dunn correction showed no pairwise significant difference. I am not surprise that these tests were not the same.

My question is how should I conclude based on these test results?

$\endgroup$
0
1
$\begingroup$

You are correct not to be surprised that the two methods give slightly different results near the 5% level. Depending on how many comparisons you made, Bonferroni-based ad hoc comparisons might be over-conservative (i.e., too "reluctant" to declare differences).

I would simply say that the main K-W test is barely significant, and give results for the most interesting ad hoc comparison(s). If there are two levels clearly with the largest difference in medians (or other quartiles) or with clearly different boxplots, then IMHO it would be OK to say that those two levels might be considered somehow different.


Of course I can't give a detailed answer without access your data and outputs from your SPSS analyses. However, consider the fictitious Likert=5 data below, for which the K-W test just barely shows overall differences (5% level) in the three locations.

set.seed(927)
x1 = sample(1:5, 72, rep=T, p = c(1,1,1,2,3))
x2 = sample(1:5, 72, rep=T, p = c(0,1,2,3,3))
x3 = sample(1:5, 72, rep=T, p = c(0,1,1,3,4))
x = c(x1,x2,x3); g = rep(1:3, each=72)

boxplot(x ~ g, horizontal=T, col="skyblue2")

enter image description here

kruskal.test(x~ g)

        Kruskal-Wallis rank sum test

data:  x by g
Kruskal-Wallis chi-squared = 6.1487, df = 2, p-value = 0.04622

Using R, I don't suppose I can do exactly the same ad hoc test you did in SPSS. However, a Wilcoxon RS test comparing levels 1 and 3 shows no significant difference at the (unadjusted) 5% level.

wilcox.test(x1,x3)$p.val
[1] 0.05116496

However, the boxplots for levels 1 and 3 look very different (even though the medians are the same), with many values in level 3 higher than values in level 1.

summary(x1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  1.000   3.000   4.000   3.667   5.000   5.000 
summary(x3)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  2.000   4.000   4.000   4.278   5.000   5.000 

Moreover, empirical CDF (ECDF) plots of these two levels show that the ECDF for level 3 lies mostly to the right of (thus below) the ECDF for level 1, suggesting stochastic dominance of level 3.

plot(ecdf(x1), col="blue", lwd=2, 
     main="ECDFs for Levels 1 (blue) and 3")
 lines(ecdf(x3), col="brown", lty="dotted", pch="o")

enter image description here

Especially if a difference between levels 1 and 3 had importance for the project at hand, I would not hesitate to mention the apparent differences between these two levels---stopping short of claiming significance.

Note: I have framed this answer in terms of 'significance' because I took that to be the point of your question. But @FrankHarrell has a point about not viewing significance with 'reverence'. The main hypothesis with P-value just barely below 5% is only weakly suggestive of differences. Then by 'logic' it would follow that some two of the three levels may be different, and 1 vs. 3 seems the best candidate. But ad hoc testing is not compelled to follow that 'logic'.

If you change the seed in the code that sampled my fictitious data, you may get fictitious data for which the K-W test is is not significant at the 5% level: 72 replications per level do not provide good power for the K-W test. (In fact, set.seed(726) leads to P-value about 18%.)

$\endgroup$
6
  • $\begingroup$ Hi there, Thank you so much for answering! I think I will just, as you suggested, write that K-W test showed borderline significance and present post-hoc P values. $\endgroup$
    – boyuan
    Sep 26 at 12:04
  • $\begingroup$ i would also like to mention that the null hypotheses for the K-W and the post-hoc tests are a bit different. So, when P is just below the significance level, 0.042 in this case, post-hoc may suggest a different answer. i would also like to mention that in my case, there are only 3 groups. Thus, in my opinion, Bonferroni corection is not that conservative. Looking forward to your opinion on this as well. $\endgroup$
    – boyuan
    Sep 26 at 12:09
  • 1
    $\begingroup$ I am alarmed at the reverence giving to "statistical significance" here. "Significance" is arbitrary and almost meaningless. I would be more interested in seeing simultaneous confidence intervals for contrasts based on a model, in this case the proportional odds model, which generalizes the Kruskal-Wallis test. This would involve odds ratios but can be converted to means, medians, or exceedance probabilities. See the Nonparametrics chapter in BBR. $\endgroup$ Sep 26 at 12:39
  • 1
    $\begingroup$ A Wilcoxon RS test is using different ranks than the Kruskal-Wallis test (the former creates ranks based only on two groups, the latter uses ranks based on $k$ groups), and is not using the pooled variance estimate implied by the Kruskal-Wallis null hypothesis, so comparing its results to a proper post hoc test (which will use the KS ranks, and a pooled variance estimate) is questionable. $\endgroup$
    – Alexis
    Sep 26 at 18:53
  • 1
    $\begingroup$ @BruceET Thanks, BruceET, great point! I have only passing acquaintance (I would not even call it familiarity) with Tukey's HSD. Do I recall aright the Tukey's HSD is appropriate when $k=3$, but that there are better methods for $k>3$? I have a vague recollection of that somewhere on CV (which I bring up because that explanation, whether valid or no, and whether I understood it correctly or no, was why I haven't studied up on T's HSD :).. $\endgroup$
    – Alexis
    Sep 26 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.