# Bonferroni correction on multiple Kruskal-Wallis tests

My data consists of 3 groups, and 6 variables. The data are not normal so I want to use non-parametric tests to uncover significance. The first thing I wanted to get your opinion on is, is it acceptable for me to run 6 Kruskal-Wallis tests but use a Bonferroni correction ($p'=p/6$). If this is acceptable, is it then reasonable to use 6 subsequent and corrected Dunn's tests to see which variables are significant?

It seems to me that this way of doing it is clunky, but I am not aware of a better way to compare my data.

P.S. I am doing this analysis in MATLAB with the statistics tool box.

• In what sense are your data not normal? How much data do you have? Are the variables understood as related (eg, measuring different aspects of the same construct)? Commented Jan 14, 2015 at 20:19
• The variables are independent in the sense that one observation can be described by these 6 variables. It's a bit hard to explain what I mean by non-normal but briefly, I used a mahalanobis distance to find observations that were outliers. I now re-grouped my data into 2 groups of these outliers and a much larger group consisting of the rest of the data. That is why now, I want to see how significantly different they are from the rest of the population(and from each other)
– Joe
Commented Jan 14, 2015 at 20:29
• That information should go in your question Commented Mar 11, 2015 at 22:43
• You might want to consider the answers to this question stats.stackexchange.com/questions/139311/… for general guidance on whether it is always sensible to correct for multiplicity of comparisons at all. Commented Mar 11, 2015 at 23:41

You would use the Bonferroni for a one-way test. But let's be clear:

You would not use the Bonferroni adjustment on the Kruskal-Wallis test itself. The Kruskal-Wallis test is an omnibus test, controlling for an overall false-positive rate.

You would use the Bonferroni for post hoc Dunn's pairwise tests. Indeed, Dunn introduced the "Bonferroni" adjustment.

You could also consider using more powerful family-wise error rate adjustment methods, such as the Holm-Sidak method, or still more-powerful false discovery rate adjustment methods, such as the Benjamini-Hochberg adjustment. These and other multiple comparisons adjustments are implemented specifically for Dunn's test in Stata (within Stata type net describe dunntest, from(https://alexisdinno.com/stata)), and in R (see http://cran.r-project.org/web/packages/dunn.test/).

You might also consider using the more powerful Conover-Iman post hoc (only) test statistic. This test is implemented for Stata (within Stata type net describe conovertest, from(https://alexisdinno.com/stata)), and for R in the conover.test package, and includes the same selection of multiple comparisons adjustment procedures.

EDIT: I missed that you seem to be wanting something like a multivariate (multiple dependent variables) nonparametric one-way ANOVA. Katz and McSweeney offer such a generalization of the Kruskal-Wallis test, although I am not aware of an implementation in software. They also provide post hoc Scheffé-like univariate and multivariate procedures, which ought to be amenable to multiple comparisons adjustments. Their test is likewise omnibus for all variables.

References
Conover, W. J. (1999). Practical Nonparametric Statistics. Wiley, Hoboken, NJ, 3rd edition.

Conover, W. J. and Iman, R. L. (1979). On multiple-comparisons procedures. Technical Report. LA-7677-MS, Los Alamos Scientific Laboratory.

Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3):241–252.

Katz, B. M. and Mcsweeney, M. (1980). A multivariate Kruskal-Wallis test with post hoc procedures. Multivariate Behavioral Research, 15:281–297.

• Thanks, I really appreciate your quick and thorough response. What I am trying to understand however is that I get how the kruskal-wallis test is omnibus for one of my 6 variables. But If I am interested in saying that the groups are different in any of the 6 variables wouldn't running multiple KW tests increase the probability of a FP in one of the variables?
– Joe
Commented Jan 14, 2015 at 20:20
• @Joe It is an omnibus test for all of your variables... just like ANOVA is an omnibus test for all variables. You perform a single KW test on all vars. Also: you are welcome. Commented Jan 14, 2015 at 20:30
• @Alexis, yes. Let me be explicitly clear, I want to find the differences between the three groups in my data. I understand that a KW followed by Dunn's post hoc test I can determine if the groups are different for one variable. But since I am also comparing 5 other variables in the same way it appears as though I would need to correct for that
– Joe
Commented Jan 14, 2015 at 21:04
• @Alexis, that's good. My impression is that female answerers are a minority on CV (I have no idea about askers). & I sometimes wonder if there might be something about CV that is unfriendly towards women's participation. As you may know, there is some discussion about unfriendliness / discrimination towards women in online forums. Commented Jan 14, 2015 at 21:16
• @beld If I understand your question, one could do independent K-W tests with post hoc tests for each DV, but if these DVs are related, then you would end up ignoring (a) the increased precision of pooled variance estimates (so your inferences would be less reliable than in a properly multivariate estimation), and (b) estimating the correlated error structure implied by a multivariate model. If multivariate structure is simply a nuisance, (b) might be less of a concern (maybe?), but (a) may still bite you. Commented Aug 24, 2022 at 18:34