I am trying to understand multiple-comparisons adjustment for confidence intervals. I came across the following links on the topic, but a few things are not clear to me.
- Stanford lecture: Section 2 on Multiple confidence intervals and Section 3 on Coverage after selection
- Cross Validated: Simultaneous and selective confidence intervals
- Cross Validated: Selected parameters and candidate parameters
All these links talk about the distinction between these two cases: (i) when a set of parameters are selected from the candidate parameters, (ii) when all candidate parameters are used. Is there a difference in how you adjust for multiple comparisons in these two cases?
Here is a concrete example:
I have done 10 regression analyses, which I have grouped into two groups of five analyses. The grouping is based on domain knowledge. Let's say group 1 is related to a certain type of plants and group 2 is related to certain other type of plants. Now within each group, I have five confidence intervals. Currently, I am doing the following steps to account for multiple comparisons:
- Select significant associations in each group, and denote the number of significant associations by s1 <= 5 (group 1) and s2 <= 5 (group 2)
- In each group, apply Bonferroni correction to construct confidence intervals for the selected associations.
Is this a valid method? Or should I apply Bonferroni correction to all five confidence intervals (instead of s1 and s2)? Additionally, is there a way to apply FDR to these confidence intervals?