5
$\begingroup$

I've used the Mann-Whitney U-test in order to check for possible differences between two relatively small groups (12 subject in each) on 15 different items. I didn't have any pre-defined assumption about which items would be different. However this simple Mann-Whitney U-test yielded a significant different (P<0.05) for ALL items tested.

My question: if/what corrections for multiple comparisons should I use here? Does the fact that I have found a significant different in all items make any difference?

Improve this question
$\endgroup$
0
3
$\begingroup$

I don't think that the fact that you have found significant differences in all fifteen of your comparisons makes a difference. To maintain the familywise error rate, I would be tempted to simply apply a Bonferroni correction. Perhaps it's good to be conservative in this instance given your small sample size (indeed, even smaller than the number of comparisons you are making) and the fact that you began the analysis with no predefined assumptions. Although if this latter fact is true, replication in an independent population would really be in order if you want to make stronger conclusions.

Improve this answer
$\endgroup$
4
$\begingroup$

Alexander's answer is very good and he makes a good suggestion. It does seem a little surprising that everything is significant when the sample size is relatively small as in your example. The Bonferroni bound may be too conservative though if some of the p-values are close to 0.05. I would suggest p-value adjustment using a bootstrap or permutation approach might do better. In SAS you can do this with PROC MULTTEST. If you are not familiar with these methods look at the text by Westfall and Young.

Improve this answer
$\endgroup$
0
$\begingroup$

This is one of many questions on here that comes at the problem of multiple comparisons backwards. Indeed, it is basically nonsensical for researchers to look at their results and then ask, "what correction for multiple comparisons is appropriate for p-values that look like this?" Researchers should decide BEFORE looking at the p-values (or at least independently of what the p-values are) what correction is appropriate for the hypotheses being tested.

Since this type of question comes up fairly frequently, perhaps we should create a single page that all such questions can be redirected to somehow.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Re your last sentence: that's how SE is supposed to work. If you find another question that's an exact duplicate of this (i.e. such that the answers here fully answer it) or vice-versa then please see How should duplicate questions be handled? for how to proceed. If you see questions that are related but not duplicates (so that answers to one would be very useful for people who asked the other -- even if they don't count as full answers) ... ctd $\endgroup$ – Glen_b Mar 5 '17 at 8:06
  • $\begingroup$ ctd ... and these related questions don't appear in each others "Related Questions" lists on the right hand sidebar, then you can comment under the question with a link to the other useful question (it will then show up under "Linked" on the sidebar). Both these actions help people find good answers. $\endgroup$ – Glen_b Mar 5 '17 at 8:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.