7
$\begingroup$

In a discussion with another researcher, I was informed that it is not always necessary to perform a Bonferroni correction on exploratory research if a lot of testing is required (for example, if many questions are asked and ANOVAs are required for each of them) as it will deviate further and further away from the 0.05 level.

Could anyone recommend any papers/references that explain this situation in detail to improve my understanding and for use in my dissertation?

Many thanks in advance!

$\endgroup$
1
  • $\begingroup$ This is a broad question. Could you narrow it by describing the kind of "exploratory research" or procedures you plan on doing? In the meantime, because it essentially asks for a list of results, I have made it CW. $\endgroup$
    – whuber
    Commented Mar 24, 2016 at 16:59

2 Answers 2

1
$\begingroup$

You should generally address the issue of multiple testing in some way. That doesn't mean Bonferroni is the best approach in all cases, however. Different methods address different error rates and the proper method depends on the goals of the testing and the consequences of making a Type I error. Try this paper:

Frane, A. V. (2015). Planned Hypothesis Tests Are Not Necessarily Exempt From Multiplicity Adjustment. Journal of Research Practice, 11(1). Available from http://jrp.icaap.org/index.php/jrp/article/view/514/417

$\endgroup$
1
$\begingroup$

The Bonferroni correction is a pretty conservative approach to hypothesis testing. For $n$ tests, it requires you a p-value of $p/n$ where $p$ is your significance level. This guarantees that the probability of you getting a positive by pure chance stays below $p$, but sometimes it makes it go way below $p$, thus also losing power in your tests.

For instance, if you want to perform a million tests, a $p$-value of $5\cdot10^{-8}$ for each of them will result in an "overall $p$-value" of much less than $0.05$

This does not mean that a $p$-value correction is not necessary when you have a lot of tests. It's quite the opposite! If you make enough comparisons, you will eventually get false positives if you do not adjust your $p$-value. The issue here is that Bonferroni "takes things too far". For a more balanced approach, you may want to use something like Tukey's honest significance

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.