A question: I have a study with 3 experiments, each involving a between-subjects comparison of a treatment vs. control group. The groups in each experiment are independent samples and no groups are repeated across experiments. A t-test is run for each experiment.

A reviewer is asking for correction for multiple comparisons. It seems odd to perform such a correction across experiments, especially since we only ran one t-test per experiment. However, in all 3 experiments we are interested in whether there is an effect of the treatment (albeit under different conditions in each case). Is Bonferroni the appropriate method here? What other methods might be preferable? Are there citeable works on whether it is appropriate to do such corrections?

Note: a somewhat similar question was asked here: correcting for multiple comparisons with independent groups

Thanks for any advice!

  • $\begingroup$ Have a look at this answer here It also provides some references that may help you in your specific situation. In general, the appropriate method will depend on your experiment and what you want to control for. $\endgroup$ – Stefan Mar 17 '18 at 13:50

A Bonferroni correction is conservative, but always an option, because the Bonferroni inequality requires no assumptions about the relationships among the events. It exerts the strongest control over the simultaneous error probability, $\Pr(\text{at least one type I error})$, unconditionally, and may be used also for constructing (conservatively) a set of confidence intervals whose simultaneous confidence level exceeds the specified one. For just 3 tests, it usually isn’t all that conservative, and so it has much to recommend it when this strong type of control over the error rate is desired.

You could also use a slightly less conservative adjustment and run each test at an individual significance level of $1-(1-\alpha)^{1/3}$ (which is about 0.017 with $\alpha=0.05$). This is based on setting the joint probability of 3 independent events to the desired value, and it is justified because you have independence here. Again, this would control the overall error probability at the simultaneous CI level.

Some other simultaneous-testing procedures gain power by avoiding testing certain comparisons at all when other comparisons are already deemed non-significant. There are various levels of control over the error rate with such procedures. Methods that control the false-discovery rate are at the weakest level of control (other than no correction at all). It is unclear how one might apply such methods to the situation described in the OP. But I suppose one possibility is to fit a model with a block effect for the three experiments, and a treatment factor that has 4 (or 6?) levels: control, treat1--treat3 (or perhaps 3 levels of control?). A Dunnett-style test could be used to test the comparisons of interest. The Dunnett method has the same strong control over error rate as the Bonferroni, and be slightly more powerful than Bonferroni -- I suspect not that much. An FDR method could be applied using something like the R function p.adjust() with the unadjusted $P$ values.

A good reference for all the issues under discussion is Chapter 5 of Oehlert's experimental design text, which is openly available by the Creative Commons license: http://users.stat.umn.edu/~gary/book/fcdae.pdf

  • $\begingroup$ The Bonferroni adjustment is always wrong in that it (a) always assumes all null hypotheses are true, even after some have been rejected and hemorrhages statistical power thereby, and (b) rejection probabilities under it are conditional on an undefined and loosey-goosey concept of "family of tests," and change when the number of tests change. The method is 55 years old, and has been supplanted by control of the false discovery rate (FDR) which suffers neither of these problems. $\endgroup$ – Alexis Mar 17 '18 at 5:53
  • $\begingroup$ There are different types of control over the simultaneous error rate. See for example Chapter 5 of Gary Oehlert’s book, users.stat.umn.edu/~gary/book/fcdae.pdf. FDR methods establish only weak control. $\endgroup$ – Russ Lenth Mar 17 '18 at 14:04
  • $\begingroup$ @Alexis Can you explain how you conclude that the Bonferroni adjustment is always wrong (maybe you want to elaborate in a separate answer?) It's a very strong statement. Is it your personal opinion or a claim that has been established in the literature? I am just curious. I was always under the impression that there is no perfect test that can be applied across all problems you may encounter. It all depends on your data, your hypotheses, and other assumptions specific to your experiment. $\endgroup$ – Stefan Mar 17 '18 at 14:05
  • 2
    $\begingroup$ @Alexis seems to be arguing that the FDR is always the rate one should control. But I don’t think many statisticians agree with that. For one thing, sometimes people want simultaneous confidence intervals and the FDR doesn’t work for that. By the way, most popular FDR methods actually use the Bonferroni inequality, but this is done stepwise with families of different sizes. $\endgroup$ – Russ Lenth Mar 17 '18 at 18:29
  • 1
    $\begingroup$ @Alexis you are obviously missing the point. You made a pretty strong claim in your first comment. I asked for references to support this. You provide references that explain the FDR method but none of these references actually suggesting to never use Bonferroni. It seems your claim is opinion-based, which is fine, but then it should be labeled as such. You don't seem to be reading the comments because nowhere I am advocating for anything. If you think you can help the OP with his/her problem, I'd suggest you post an answer since answers on CV can also be cited! $\endgroup$ – Stefan Mar 18 '18 at 18:35

Dunnetts multicomparrison test is used for multiple comparisons when there is a control

For reference: http://www.statisticshowto.com/dunnetts-test/

  • 1
    $\begingroup$ I disagree. Dunnett’s method applies when you have one control and several other treatments in ONE experiment. $\endgroup$ – Russ Lenth Mar 17 '18 at 2:39
  • $\begingroup$ It sounds like they should have done an anova based on reveiwer comment . Almost as if each experiment is really a different factor or factor level. So I treated it like posthoc tests. If they strongly disagree with the reveiwer and they truly are independent experiments they should write a rebuttal. $\endgroup$ – OliverFishCode Mar 17 '18 at 2:48
  • 1
    $\begingroup$ Thanks for the thoughts. An ANOVA across all experiments might be worth considering. We initially already sidestepped the issue in our first invited revision, but this is the second round, and the reviewer is more insistent this time (yet rather vague, only stating correcting for multiple comparisons). $\endgroup$ – Altair555 Mar 17 '18 at 6:42
  • $\begingroup$ I can't comment on the other answers, but here is a discussion of bonferoni from another question: stats.stackexchange.com/questions/120362/… $\endgroup$ – OliverFishCode Mar 17 '18 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.