# multiple comparison with controls - kruskall wallis or mann whitney + bonferroni

I have a test group and two control groups. I would like to compare:

test vs control group 1

test vs control group 2

and I don't need the result from control group 1 vs group 2. Therefore, in this case, should I use Mann Whitney U with $$p$$ x $$2$$ for Bonferroni correction or should I use Kruskal Wallis test?

• I wonder about your lack of curiosity whether the two control groups differ. If the test group differs significantly from C1, but not from C2, will you claim a 'success'? Or are you hoping the test group differs signif from both C1 and C2? // You must have thought C1 & C2 differ in some way. Otherwise, why bother with both? // Is this for a protocol in advance of the study or have you already seen the data? – BruceET Nov 18 '18 at 23:10
• @bruceET, this is because the difference between the two control group was already quantified in a previous analysis, and the present analysis is just an extension to that. This is not protocoled in advance I have seen the data. I could do all the analysis together instead of separating into two parts but it would distrupt the story. – Sharah Nov 20 '18 at 17:37
• This reads like a standard ANOVA application. Could you indicate how it differs? – whuber Nov 20 '18 at 17:46
• @whuber the difference is that between these three sets of data, i am only interested to see the effect between two pairs of data instead of all three pairs. – Sharah Nov 23 '18 at 12:41

An appropriate post-hoc test for Kruskal-Wallis is Dunn (1964) test. You could use Dunn test, look at only the comparisons you are interested in, and use a p-value correction † based on only those comparisons you are interested in.

There are other appropriate post-hoc tests.

The problem with using multiple Mann-Whitney tests is that each will be discarding information from the other groups. Results can be incommensurate or difficult to interpret. (See Schwenk dice.)

Dunn test essentially keeps the ranking of values from the original Kruskal-Wallis and avoids this problem. This is analogous to using pairwise t-tests with pooled variance from all groups. (See, for example, how R conducts pairwise t-tests.)

† In most cases, Bonferroni correction is overly conservative. You might consider Holm, or other methods used natively by R. See the Details section at that link for a list and very brief discussion.

• I am using MATLAB instead of R and I am not clear about "use a p-value correction † based on only those p-value corrections you are interested in" what do you mean by interested in? In MATLAB, for eg, there will be only 1 p-value for each pair of test uk.mathworks.com/help/stats/multiple-comparisons.html – Sharah Nov 23 '18 at 12:40
• I wrote that slightly incorrectly. I'll correct. But I mean: You have three groups total, so if you looked at all comparisons, you would have three comparisons. If you were adjust alpha by Bonferroni, your corrected alpha would be 0.05 / 3 = 0.0167. But you want to look at only two comparisons, so the alpha by Bonferroni would be 0.05 / 2 = 0.025. You can ignore the comparison of Control 1 and Control 2 if that's what you want to do. – Sal Mangiafico Nov 23 '18 at 12:54
• I see, the only thing is, see you see the link given above, in MATLAB, the correction is already performed in the function. therefore the P value is the corrected values already, assumingly according to the number of pairs – Sharah Nov 23 '18 at 13:33
• It's not clear to me what post-hoc test MatLab uses for Kruskal-Wallis. But if the software is adjusting p-values with Bonferroni, it's pretty easy to reverse the correction. If there are three comparisons, simply divide the reported p-values by 3 to get the uncorrected p-values. Then you can adjust the two p-values you keep with Bonferroni by hand. Or, if you would like to then use a different correction on the p-values you keep, you can run R online here, with e.g. this code: Pvalues = c(0.002, 0.03); p.adjust(Pvalues, method="holm"). – Sal Mangiafico Nov 23 '18 at 14:04