On the corrections for multiple comparisons

Question

I've used the Mann-Whitney U-test in order to check for possible differences between two relatively small groups (12 subject in each) on 15 different items. I didn't have any pre-defined assumption about which items would be different. However this simple Mann-Whitney U-test yielded a significant different (P<0.05) for ALL items tested.

My question: if/what corrections for multiple comparisons should I use here? Does the fact that I have found a significant different in all items make any difference?

Answer 1

I don't think that the fact that you have found significant differences in all fifteen of your comparisons makes a difference. To maintain the familywise error rate, I would be tempted to simply apply a Bonferroni correction. Perhaps it's good to be conservative in this instance given your small sample size (indeed, even smaller than the number of comparisons you are making) and the fact that you began the analysis with no predefined assumptions. Although if this latter fact is true, replication in an independent population would really be in order if you want to make stronger conclusions.

Answer 2

Alexander's answer is very good and he makes a good suggestion. It does seem a little surprising that everything is significant when the sample size is relatively small as in your example. The Bonferroni bound may be too conservative though if some of the p-values are close to 0.05. I would suggest p-value adjustment using a bootstrap or permutation approach might do better. In SAS you can do this with PROC MULTTEST. If you are not familiar with these methods look at the text by Westfall and Young.

Answer 3

This is one of many questions on here that comes at the problem of multiple comparisons backwards. Indeed, it is basically nonsensical for researchers to look at their results and then ask, "what correction for multiple comparisons is appropriate for p-values that look like this?" Researchers should decide BEFORE looking at the p-values (or at least independently of what the p-values are) what correction is appropriate for the hypotheses being tested.

Since this type of question comes up fairly frequently, perhaps we should create a single page that all such questions can be redirected to somehow.