1
$\begingroup$

I am trying to understand multiple-comparisons adjustment for confidence intervals. I came across the following links on the topic, but a few things are not clear to me.

All these links talk about the distinction between these two cases: (i) when a set of parameters are selected from the candidate parameters, (ii) when all candidate parameters are used. Is there a difference in how you adjust for multiple comparisons in these two cases?

Here is a concrete example:

I have done 10 regression analyses, which I have grouped into two groups of five analyses. The grouping is based on domain knowledge. Let's say group 1 is related to a certain type of plants and group 2 is related to certain other type of plants. Now within each group, I have five confidence intervals. Currently, I am doing the following steps to account for multiple comparisons:

  • Select significant associations in each group, and denote the number of significant associations by s1 <= 5 (group 1) and s2 <= 5 (group 2)
  • In each group, apply Bonferroni correction to construct confidence intervals for the selected associations.

Is this a valid method? Or should I apply Bonferroni correction to all five confidence intervals (instead of s1 and s2)? Additionally, is there a way to apply FDR to these confidence intervals?

Improve this question
$\endgroup$
5
  • $\begingroup$ Some views on the Bonferroni correction here: stats.stackexchange.com/questions/120362/… $\endgroup$
    – fblundun
    Commented Dec 17, 2020 at 19:52
  • $\begingroup$ I appreciate the limitations of the Bonferroni correction, and have been aware of the discussion. However, for good or bad, a correction method is expected in many research areas when one tries to report findings that use multiple comparison. If there is a better alternative for confidence intervals, I'd love to know more. $\endgroup$
    – vbip
    Commented Dec 18, 2020 at 5:55
  • $\begingroup$ Is your step 1 selection going to be done using a multiplicity correction? If not, and if part of your published result is going to be claiming that each association found in step 1 is significant, then when the null hypotheses are true your probability of reporting at least one false positive ends up being $> \alpha$ regardless of any multiplicity correction done in creating confidence intervals. $\endgroup$
    – fblundun
    Commented Dec 18, 2020 at 22:46
  • $\begingroup$ As for other methods of multiplicity correction for confidence intervals, a couple are discussed here stats.stackexchange.com/questions/144807/…, but it seems like they might only be intended for confidence intervals for the null hypothesis rather than confidence intervals for parameters estimated from data. $\endgroup$
    – fblundun
    Commented Dec 18, 2020 at 22:48
  • $\begingroup$ Hi @fblundun, could you please elaborate a bit? In step 1, the confidence intervals are used to test the null hypotheses (that the confidence intervals do not contain zero). I run five regressions (in each group) and build the confidence intervals from the estimated coefficients and standard errors. Now before reporting the results, I want to do multiple comparison correction. Can I select the significant confidence intervals and do the correction for them only? $\endgroup$
    – vbip
    Commented Dec 19, 2020 at 4:25

1 Answer 1

Reset to default
1
$\begingroup$

The point of the Bonferroni correction is to limit the probability of reporting at least one false positive to $\le \alpha$, regardless of which null hypotheses are true. I think that doing the correction only for the confidence intervals that you have found to be significant does not achieve this limit, so is hard to justify.

Suppose your 5 null hypotheses are all true. And suppose you pick $\alpha = 0.05$. Then the probability of getting exactly one false positive is $0.05 \cdot0.95^{4}\cdot5 > 0.2$ (assuming the 5 analyses are independent). In this case your $s_1$ is only 1, so applying Bonferroni correction to this single supposedly significant association doesn't make a difference. You then build a "95%" confidence interval for the true value of this association based on your data. This confidence interval fails to contain the null hypothesis. So with probability > 20%, you end up reporting a confidence interval which doesn't contain the true value.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for the reply, and my precise reason for confusion is this "selective" vs "all" comparisons. I found a paper where the language seems to suggest that Bonferroni correction was done only for significant associations (see Caption of Figure 2 in this paper bmj.com/content/367/bmj.l6258). What do you think is going on? This SE discussion also mentions "simultaneous coverage of selected parameters" but I'm not sure I understand what it is recommending one should do stats.stackexchange.com/a/132610/26955. $\endgroup$
    – vbip
    Commented Dec 20, 2020 at 20:12
  • $\begingroup$ I just don't see the philosophical justification for choosing the Bonferroni correction factor used for widening confidence intervals based on the number of significant associations. What's the point in doing a multiplicity correction if it doesn't bound the FWER? $\endgroup$
    – fblundun
    Commented Dec 20, 2020 at 23:01
  • $\begingroup$ I'm not sure. Is it based on the number of comparisons? We are comparing if the mean of the coefficient is different than zero for only those confidence intervals which are interesting. My reading of the following lecture (Section 3) is that multiplicity correction is only required when one is selecting interesting confidence intervals. the SE answer I linked above seems to be making the same argument. Would be great to understand this in more detail. statweb.stanford.edu/%7Ecandes/teaching/stats300c/Lectures/… $\endgroup$
    – vbip
    Commented Dec 21, 2020 at 3:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.