I have a question about adjusting pvalues to account for multiple testing, as I need to do this given a small sample size with a lot of tests.

I have read that the most common and conservative method of controlling for the family-wise error rate is to use the bonferroni method (alpha / k, where k is the number of tests). I can also do this doing p.adjust on R. However, I have a question about k. Should k in this case be the number of comparisons overall, or should it be adjusted by the type of test? In other words, I have 8 t-tests, 8 chi-squares, and 8 Mann-Whitney tests. Should I do alpha/8, for each, or alpha/24?

This may be a simple question, but it has really gotten me stuck! I know the bonferroni correction is typically done for ANOVAs and it's done within each test, not for all other tests, so it is somewhat confusing.

Thanks for your help!



You divide the p-value by the number of comparisons you want to make, thus you should divide the alpha value you defined by 24.

"The Bonferroni correction is a commonly-used approach for addressing the multiple comparisons problem, partly because of its simplicity. If you conduct k tests, the target significance level should be alpha/k, or, equivalently, you multiply your p-values by k, and apply the standard alpha level. (The trouble with multiplying the p-values is sometimes you end up with values over one, rendering the interpretation of the p-values incoherent.)

For example, suppose you conduct an experiment that has 3 dependent variables. You conduct three difference-in-means tests that yield the following classical p-values: 0.004, 0.020, and 0.122. If your alpha level is the standard 0.05 threshold, then you would usually declare the first two tests statistically significant and the last test insignificant. The Bonferroni correction, however, adjusts the target p-value to 0.05/3 = 0.016. We then declare only the first test to be statistically significant."

I hope this answers your question. You can find the information at http://egap.org/methods-guides/10-things-you-need-know-about-multiple-comparisons


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.