As background, multiple comparison tests (MCT) are a new topic for me. I am currently planning the data analysis for an experiment and see that an MCT is needed. It seems that a key differentiating factor in MCTs concerns their statistical power. In this experiment, rejecting many (even all) null hypotheses rather than failing to reject, would actually be more consistent with the theoretical predictions I am testing. Thus, it seems that choosing the most powerful test to reduce the probability of Type II errors may in some way be "stacking the deck" in favor of confirming the theoretical predictions. Yet, I wonder if being as conservative as possible (i.e. using the Bonferroni Correction) may also be extreme.
Is this an issue to address? Is there a principled way to try and balance this tension? What is a "reasonable" amount of statistical power to apply in such a situation?
A few more possibly pertinent details:
- The number of hypotheses will probably be at most 500. Probably far fewer, 50 to 100, seems more likely.
- It seems that many MCTs are options--but given my lack of background I could be wrong about that. For instance, obviously the conservative tests can be applied, I also anticipate positive dependence between the hypotheses (i.e. if one is rejected, it is more likely another is rejected) which opens up another class of tests.
Many thanks in advance for your help!