# Multiple comparison correction for confidence interval

I am trying to understand multiple-comparisons adjustment for confidence intervals. I came across the following links on the topic, but a few things are not clear to me.

All these links talk about the distinction between these two cases: (i) when a set of parameters are selected from the candidate parameters, (ii) when all candidate parameters are used. Is there a difference in how you adjust for multiple comparisons in these two cases?

Here is a concrete example:

I have done 10 regression analyses, which I have grouped into two groups of five analyses. The grouping is based on domain knowledge. Let's say group 1 is related to a certain type of plants and group 2 is related to certain other type of plants. Now within each group, I have five confidence intervals. Currently, I am doing the following steps to account for multiple comparisons:

• Select significant associations in each group, and denote the number of significant associations by s1 <= 5 (group 1) and s2 <= 5 (group 2)
• In each group, apply Bonferroni correction to construct confidence intervals for the selected associations.

Is this a valid method? Or should I apply Bonferroni correction to all five confidence intervals (instead of s1 and s2)? Additionally, is there a way to apply FDR to these confidence intervals?

• Some views on the Bonferroni correction here: stats.stackexchange.com/questions/120362/… Dec 17, 2020 at 19:52
• I appreciate the limitations of the Bonferroni correction, and have been aware of the discussion. However, for good or bad, a correction method is expected in many research areas when one tries to report findings that use multiple comparison. If there is a better alternative for confidence intervals, I'd love to know more.
– vbip
Dec 18, 2020 at 5:55
• Is your step 1 selection going to be done using a multiplicity correction? If not, and if part of your published result is going to be claiming that each association found in step 1 is significant, then when the null hypotheses are true your probability of reporting at least one false positive ends up being $> \alpha$ regardless of any multiplicity correction done in creating confidence intervals. Dec 18, 2020 at 22:46
• As for other methods of multiplicity correction for confidence intervals, a couple are discussed here stats.stackexchange.com/questions/144807/…, but it seems like they might only be intended for confidence intervals for the null hypothesis rather than confidence intervals for parameters estimated from data. Dec 18, 2020 at 22:48
• Hi @fblundun, could you please elaborate a bit? In step 1, the confidence intervals are used to test the null hypotheses (that the confidence intervals do not contain zero). I run five regressions (in each group) and build the confidence intervals from the estimated coefficients and standard errors. Now before reporting the results, I want to do multiple comparison correction. Can I select the significant confidence intervals and do the correction for them only?
– vbip
Dec 19, 2020 at 4:25

The point of the Bonferroni correction is to limit the probability of reporting at least one false positive to $$\le \alpha$$, regardless of which null hypotheses are true. I think that doing the correction only for the confidence intervals that you have found to be significant does not achieve this limit, so is hard to justify.
Suppose your 5 null hypotheses are all true. And suppose you pick $$\alpha = 0.05$$. Then the probability of getting exactly one false positive is $$0.05 \cdot0.95^{4}\cdot5 > 0.2$$ (assuming the 5 analyses are independent). In this case your $$s_1$$ is only 1, so applying Bonferroni correction to this single supposedly significant association doesn't make a difference. You then build a "95%" confidence interval for the true value of this association based on your data. This confidence interval fails to contain the null hypothesis. So with probability > 20%, you end up reporting a confidence interval which doesn't contain the true value.