# How to select a Multiple Comparisons Test based on power?

As background, multiple comparison tests (MCT) are a new topic for me. I am currently planning the data analysis for an experiment and see that an MCT is needed. It seems that a key differentiating factor in MCTs concerns their statistical power. In this experiment, rejecting many (even all) null hypotheses rather than failing to reject, would actually be more consistent with the theoretical predictions I am testing. Thus, it seems that choosing the most powerful test to reduce the probability of Type II errors may in some way be "stacking the deck" in favor of confirming the theoretical predictions. Yet, I wonder if being as conservative as possible (i.e. using the Bonferroni Correction) may also be extreme.

Is this an issue to address? Is there a principled way to try and balance this tension? What is a "reasonable" amount of statistical power to apply in such a situation?

A few more possibly pertinent details:

• The number of hypotheses will probably be at most 500. Probably far fewer, 50 to 100, seems more likely.
• It seems that many MCTs are options--but given my lack of background I could be wrong about that. For instance, obviously the conservative tests can be applied, I also anticipate positive dependence between the hypotheses (i.e. if one is rejected, it is more likely another is rejected) which opens up another class of tests.

• (1) If by 'nr of hypotheses' you mean 500 potential ad hoc pairwise comparisons among gps you must have about $g = 32$ gps (levels of factor). (2) If your gps are from normal populations with equal variances, and you have $r$ replications in each grp, then Tukey HSD would use all $gr$ obs to est common SD $\sigma,$ thus giving good power even after 'false discovery' protection. (3) Not "stacking the deck" to have $r$ suff large for good power to detect real diffs of important size. Good power will not artificially create bogus diffs. // Too many issues for one Q. Pls focus/clarify. Commented Aug 13, 2020 at 20:45
• @BruceET Thanks! I hope this is clarifying: (1) Yes, it will be something like 500 potential ad hoc pairwise comparisons, but not necessarily between every pair of groups, but think that's a minor point since (2) Tukey HSD seems not to apply as I seek to apply many two-sample tests, specifically, the Maximum Mean Discrepancy, since the data are high-dimensional up to d = 1,000 and I can't make any assumptions on an analytical form of the distribution. (3) Please excuse my ignorance, I'm not sure what you mean by replications: Bootstrapping? Or actual fresh samples? Commented Aug 13, 2020 at 21:25
• Replications $r$ is the number of observations at each level of the factor. Commented Aug 13, 2020 at 22:01