0
$\begingroup$

As background, multiple comparison tests (MCT) are a new topic for me. I am currently planning the data analysis for an experiment and see that an MCT is needed. It seems that a key differentiating factor in MCTs concerns their statistical power. In this experiment, rejecting many (even all) null hypotheses rather than failing to reject, would actually be more consistent with the theoretical predictions I am testing. Thus, it seems that choosing the most powerful test to reduce the probability of Type II errors may in some way be "stacking the deck" in favor of confirming the theoretical predictions. Yet, I wonder if being as conservative as possible (i.e. using the Bonferroni Correction) may also be extreme.

Is this an issue to address? Is there a principled way to try and balance this tension? What is a "reasonable" amount of statistical power to apply in such a situation?

A few more possibly pertinent details:

  • The number of hypotheses will probably be at most 500. Probably far fewer, 50 to 100, seems more likely.
  • It seems that many MCTs are options--but given my lack of background I could be wrong about that. For instance, obviously the conservative tests can be applied, I also anticipate positive dependence between the hypotheses (i.e. if one is rejected, it is more likely another is rejected) which opens up another class of tests.

Many thanks in advance for your help!

Improve this question
$\endgroup$
3
  • $\begingroup$ (1) If by 'nr of hypotheses' you mean 500 potential ad hoc pairwise comparisons among gps you must have about $g = 32$ gps (levels of factor). (2) If your gps are from normal populations with equal variances, and you have $r$ replications in each grp, then Tukey HSD would use all $gr$ obs to est common SD $\sigma,$ thus giving good power even after 'false discovery' protection. (3) Not "stacking the deck" to have $r$ suff large for good power to detect real diffs of important size. Good power will not artificially create bogus diffs. // Too many issues for one Q. Pls focus/clarify. $\endgroup$
    – BruceET
    Commented Aug 13, 2020 at 20:45
  • $\begingroup$ @BruceET Thanks! I hope this is clarifying: (1) Yes, it will be something like 500 potential ad hoc pairwise comparisons, but not necessarily between every pair of groups, but think that's a minor point since (2) Tukey HSD seems not to apply as I seek to apply many two-sample tests, specifically, the Maximum Mean Discrepancy, since the data are high-dimensional up to d = 1,000 and I can't make any assumptions on an analytical form of the distribution. (3) Please excuse my ignorance, I'm not sure what you mean by replications: Bootstrapping? Or actual fresh samples? $\endgroup$
    – roland
    Commented Aug 13, 2020 at 21:25
  • $\begingroup$ Replications $r$ is the number of observations at each level of the factor. $\endgroup$
    – BruceET
    Commented Aug 13, 2020 at 22:01

1 Answer 1

Reset to default
0
$\begingroup$

The main decision you have to make is what type of error you want to protect yourself from.

Do you want to protect yourself from making any false-positive errors? Then you must control the family-wise error rate (FWER), the probability that you would find a false positive if there were no true difference.

Alternatively, do you want to protect yourself from missing true-positive results so that you are willing to accept some false-positive findings? Then you must control the false discovery rate (FDR), the fraction of positive findings that is likely to be incorrect.

That's a decision only you can make, based on your knowledge of the subject matter and the standards of your field. For studies in biology that look at expression levels of 10,000 to 20,000 genes, for example, the FDR is the standard choice.

Once that decision is made, you will note that, within each class of control, the differences among the various tests aren't that substantial. There have been refinements over time, which you should look into once you've made the FWER/FDR choice. For example there is no longer a reason to use the original Bonferroni correction because its modification by Holm provides the same FWER control with higher power.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.