Suppose I am planning to run some well-powered factorial experiment based on a random sample with several treatments and several levels. I am interested in the effects of all of all levels vs. some baseline for each treatment. One can fit a regression model with dummy variables for all levels of all treatments (except baselines) to obtain an unbiased estimate of these "ATEs". This setting also justifies standard assumptions for a neymanian estimator of the variance and one can apply two sided t-tests for all treatment indicators (for simplicity, H0 of no effect, alpha 0.05). Naturally, a multiple comparisons problem will arise.

What is a princepled way to approach it and incorporate it in design analyses prior to conducting a study?

Bonferroni and other procrustean alpha corrections assume extreme loss functions and will likely do more harm than good. How would one approach the issue from a bayesian perspective?

Links to relevant literature are also greatly appreciated (I am aware of several bio stats papers, e.g. by Sander Greenland, but none of these were particulary eluminating, offering only vague guidance a la: "context specific loss functions have to be considered" but not how to do this)!

With the example above, I wanted to set up a case were the interest really lies in testing many different hypotheses for many causally identified parameter estimates. Often, the debate drifts into discussions about what relevant comparisons are.

  • $\begingroup$ Adjustment for multiple testing is often not necessary or desirable as it depends on the nature of inferences and context. You give no detail of context or the types of inferences that you wish to be able to make. That makes answering your question impossible. Have you looked at the many other questions on this site related to the topic? $\endgroup$ Oct 25, 2020 at 20:31
  • $\begingroup$ I have read many diverging opinions on this. I wanted to give an abstract context for a setting where I am actually interested in testing many hypotheses (i.e. I am interested in the effects of many independently randomized treatements). Many arguments I read were related to hierachical models, effect heterogeneity etc. $\endgroup$
    – persephone
    Oct 26, 2020 at 11:27
  • $\begingroup$ Fictional example (will move into question if you find it clear): Suppose a company wants to fine-tune a service they offer - for instance some health insurance plan. Assume that a representative sample of the relevant population is available. They run an experiment manipulating many factors (e.g. coverage which is a treatment/factor with multiple levels) of a fictional healthcare plan and let respondents indicate favorability/or let them choose between plans. The company is then interested in the average effects of all levels of the many factors (against some base-line levels). $\endgroup$
    – persephone
    Oct 26, 2020 at 11:43
  • $\begingroup$ Of course, this is maybe not the most realistic setting. For insurers, effect heterogenity/willingness to pay likely matters more. $\endgroup$
    – persephone
    Oct 26, 2020 at 11:56
  • $\begingroup$ I have read many question on this site regarding multiple comparisons. Many of them developed into philosophical debates very quickly. $\endgroup$
    – persephone
    Oct 26, 2020 at 12:01

2 Answers 2


Suppose I am planning to run some well-powered factorial experiment based on a random sample with several treatments and several levels. I am interested in the effects of all of all levels vs. some baseline for each treatment.

First, unless you took the potential multiple-comparison issue into account in design, you wouldn't really know that you had a "well-powered factorial experiment" as the power can only be evaluated in the context of how you are controlling for Type-I (false-positive) errors: that is, how you intend to correct for multiple comparisons.

Second, as your conversation with Michael Lew in comments indicates, factors other than controlling Type-I errors might be much more important in practical application. For example, in large-scale discovery studies in biosciences one typically wants to control the false-discovery rate (FDR, the fraction of false-positive findings) rather than the family-wise error rate (FWER, the probability that any of your findings is a false positive). How many potentially important discoveries do you want to throw away, just to make sure that you (probably) aren't making any mistakes? You are always making tradeoffs based on relative costs and benefits of false and true discoveries in these analyses, although those tradeoffs are too frequently hidden or unconscious.

Third, even if you want to control FWER, the Bonferroni method is too strict. The Holm modification is more powerful without jeopardizing FWER. You also can consider the Tukey HSD test for examining all pairwise comparisons, if its assumptions are reasonably met.

Fourth, as your primary comparisons are of multiple treatments against a baseline or control, you can use Dunnett's test that was designed specifically for such comparisons.

Fifth, if the "several levels" of the treatments are quantitative, you could model each treatment as a continuous predictor based on its specified quantitative levels. In particular:

  1. If outcome is linearly related to the levels, instead of estimating a large number of coefficients for each treatment (one less than the number of levels) you would represent the effect of the treatment with a single coefficient. That would not only cut down on the multiple-comparison problem but also allow for estimates at intermediate treatment levels.
  2. Even if a simple linear relationship with outcome doesn't hold, you could model quantitative treatment levels with splines or handle the treatment levels as ordinal predictors, taking their natural quantitative ordering into account. Those approaches also can cut down on the number of multiple comparisons, as you are still modeling fewer coefficients than if you treat the levels as unordered factors.
  • $\begingroup$ I wanted set up a situation where I am aiming at a well powered study and want to consider the Mc problem in some a priori design/power analysis. But yea the sentence is a bit ambiguous. Anyways, great answer! $\endgroup$
    – persephone
    Oct 29, 2020 at 7:06

On the Bayesian side of things, loss functions are certainly important: If the cost of a false negative is high, eg, in adverse events for a vaccine trial, then you tend to not do classical multiple comparisons. Indeed, you don't even do regular (unadjusted) statistical inference; the current COVID-19 vaccine trial haltings are an excellent case in point.

But one must not overlook priors, as they have an enormous impact upon whether one should perform classical multiple comparisons. In the case where there is prior reason to believe that many or all of the null models are all (nearly) true, then there is good reason to believe that unadjusted testing will yield false positives. For example, there is a cottage industry in manufacturing "me too" drugs in the pharmaceutical world: One company patents a molecule that works, then others rush in to patent other molecules that are similar with the exception of a moved atom here or there. It is a different chemical, but the moved atom may have no substantive effect. So in a (pre-clinical) screening experiment involving several such compounds, there is good reason to expect that one or more of such chemicals will give an incorrect indication of improvement over the standard, unless multiple comparisons adjustments are applied.

The problem with some very prominent Bayesians' approach to this problem is that they completely dismiss the plausibility of near nulls. Instead, they opt for convenient hierarchical priors, usually normal, and blithely assume that shrinkage solve the problem. Such priors automatically assume that there is near zero plausibility for the near nulls. This practice violates not only Bayesian principles, but it also violates scientific principles, and it can easily lead to incorrect inferences. I won't name names, but you know who you are.

As a matter of fact, if there is prior plausibility on the collection of (near) null models, then the Bayesian posterior probabilities behave like Bonferroni adjusted p-values; see

Westfall, P.H., Johnson, W.O. and Utts, J.M.(1997). A Bayesian Perspective on the Bonferroni Adjustment, Biometrika 84, 419–427.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.