# Correct way of applying Bonferroni correction

We are planning an experiment to study the effects of new drugs on animal models of epilepsy. The experiment will be in run in 2 phases.

The hypothesis in Phase 1 is: "Animals receiving Drug X experience a reduction in seizures." Experimental design: One group of control animals and 5 groups of test animals, each receiving a different test drug. Hence, 5 separate comparisons of Test Drug v Control.

The two best drugs from Phase 1 will be taken into Phase 2. In Phase 2, the two test drugs are withdrawn and animals are observed for any persisting benefit.

The hypothesis in Phase 2 is: "Animals that have received Drug X in the past continue to experience enduring benefit." Experimental design: One control group of animals and 2 test groups, each having received a test drug in the past. Hence, 2 separate comparisons of Test Drug vs Control.

Any drug effective in Phase 2 will be taken into human clinical trials. If no drug is effective in Phase 2, then any drug(s) effective in Phase 1 will be taken into human clinical trials. Rationale: any drug which produces enduring benefit even after stopping it would be of greatest interest. However, any drug that is effective while being administered is still of interest. (Additional detail added in response to initial replies.)

How should Bonferroni correction be applied? Separately for each phase , i.e., 0.01 (=0.05/5) for Phase 1 and 0.025 (=0.05/2) for Phase 2? Or 0.007 (=0.05/7) across both phases?

Thank you!

• Surely there is no point in applying any form of 'correction' for multiplicity in the first experiment. How would a Bonferroni procedure alter the outcome? – Michael Lew Apr 1 '17 at 20:51
• Researcher, @Nasif please merge your accounts – Glen_b Apr 2 '17 at 5:55

When comparing groups to a control group the Bonferroni correction over corrects. The Dunnett's test has more power while still controlling the Type I error rate.

The Bonferroni corrects for the case in which the probability of a Type I error is highest (independent comparisons). Dunnett's test takes into account the fact that comparisons of experimental groups with the control group will be correlated and thus "corects" less.

• I added some context. I believe others have addressed the one versus two stage issue. – David Lane Apr 2 '17 at 15:01

As Michael Lew notes in a comment, if you are going to take the two best drugs from Phase 1 in any event then a Bonferroni or other multiple-comparison test would not affect what you do. You should of course compare the 5 treatments carefully but your point then is to see, for example, how certain you are that you really found the best 2 out of 5 drugs. For that, issues like confidence intervals are critical and multiple testing corrections are not really at issue.

To many practitioners, a small number of pre-planned comparisons as you propose for Phase 2 do not require multiple-comparison corrections. Those corrections are most important when you have no pre-defined hypothesis and your hypotheses for comparisons were developed post hoc, based on the results that you got. If you (or reviewers) nevertheless insist on doing a multiple-testing correction, David Lane is correct that Bonferroni is typically much too conservative. In any event, the correction would only be applied to the 2 cases in Phase 2.

• Multiple testing is always an issue in these circumstances. – Michael R. Chernick Apr 1 '17 at 23:44

The first question to me is whether type I error rate control makes sense here, given that you really are trading of different losses (i.e. if you wrongly take a drug forward, you may needlessly expose human subjects and waste research resources, while if you wrongly stop a good drug, then you are potentially giving up on a possibly promising drug) and whether a chosen power & type 1 error rate do this in a good manner is questionable.

Secondly, if type I error rate control is what you want, then as pointed out by others for the first part you can exploit that you know the correlation between the test statistics and use e.g. Dunnett's test. With 5 groups your chance of wrongly convicing yourself that one is effective when it is in fact not is otherwise pretty high, if you do not have some form of control for this. It seems logical to first test the Phase 1 hypotheses ignoring the Phase 2 hypotheses.

The more difficult part is how to do the Phase 2 hypotheses. In particular, whether you can see that as two null hypotheses being tested, or whether that is really 5 null hypotheses - i.e. including the ones for the drugs that were not continued, for which the null hypotheses cannot be rejected, but you need to correct for their presence, anyway. In the latter case, testing the null hypothesis regarding "enduring benefit" with a Dunnett test at level $n \times \alpha/5$ across the $n$ drugs, for which the Phase 1 null hypothesis was rejected, may make sense. There should be plenty of literature on this question in e.g. the literature on "play-the-winner" or adaptive designs.