Currently I'm thinking about using confidence intervals to compare the difference in means between a few groups. However, if I understand correctly I need to use the Bonferroni correction if I test multiple hypothesis with the same data set. Obviously, this means that my confidence interval becomes more strict. I wonder if the Bonferroni correction is really neccessary. I read papers that discourage using the Bonferroni correction and one should rather take a look on effect sizes to interpret the results.

Are they correct? Is the Bonferroni correction really overrated and one should rather use effect sizes to interpret the results?

  • $\begingroup$ What field is this in? I don't see many statisticians using effect sizes. Also, p-value adjustment has no effect on the SE / CI, only on the p-values. Instead of Bonferroni use something like Holm, FDR, ... $\endgroup$ – user2974951 Aug 6 '19 at 12:41
  • 2
    $\begingroup$ @user2974951 not true. Only a bad statistician would completely disregard the effect size. Statistical significance does not imply significance in the specific domain. A drug that lowers the heart rate by 0.1 BPM might be statistically significant, but no sane company would put it into production, because it is not clinically significant. Regarding Bonferroni correction, of course this can be applied to CIs. The correction is perfomed by either taking p-value times n or divide alpha by n. Thus, you can construct the CI on the corrected alpha. $\endgroup$ – Scholar Aug 6 '19 at 12:58
  • $\begingroup$ @bi_scholar I may have misunderstood his meaning of effect size, I was thinking of something like Cohen's d or similar. I agree CI size is even more important than p-values, if this is what we refer to "effect size". In your example I believe you are referring to the difference in statistical / practical significance, which is also true. Also, Bonferroni can be used to build a CI, although it is a hackish way to do it, I think mostly it is used for p-values. $\endgroup$ – user2974951 Aug 6 '19 at 13:16
  • $\begingroup$ @user2974951: it is true that multiple testing corrections are rarely applied to confidence intervals, but this is mainly due to a misunderstanding of the concept. As you certainly know, the idea is to control the overall type I error rate. Therefore, not the p-values should be changed, but the level of alpha per test. Whether each test uses p-values or confidence intervals should not matter at all, the change in alpha stays the same. $\endgroup$ – LuckyPal Aug 6 '19 at 13:26
  • $\begingroup$ So if I understand you correctly there is no way around alpha level correction in CI? $\endgroup$ – Brain Damage Aug 6 '19 at 17:24

If you wish to control the familywise type I error rate, then you need to adjust for multiplicity. In particular, if you wish to emphasize p-value, statistical significance or whether CIs exclude some value, to claim that some of multiple comparisons you are looking at are "statistically significant", then that is often a situation where that might be something you wish to do.

The Bonferroni correction is one of the most simple (and most conservative) adjustments for multiplicity. You can easily adjust your CIs to match the adjustment (e.g. with 2 hypotheses you calculate 97.5% confidence intervals instead of 95% CIs). Other adjustments are uniformly more powerful (e.g. Bonferroni-Holm), but make it hard to find matching CIs.

There are of course approaches for dealing with multiplicity other than controlling the familywise type I error rate, e.g. using shrinkage in Bayesian hierarchical models instead.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.