Suppose we have a multiple comparisons scenario such as post hoc inference on pairwise statistics, or like a multiple regression, where we are making a total of $$m$$ comparisons. Suppose also, that we would like to support inference in these multiples using confidence intervals.

1. Do we apply multiple comparison adjustments to CIs? That is, just as multiple comparisons compel a redefinition of $$\alpha$$ to either the family-wise error rate (FWER) or the false discovery rate (FDR), does the meaning of confidence (or credibility1, or uncertainty, or prediction, or inferential... pick your interval) get similarly altered by multiple comparisons? I realize that a negative answer here will moot my remaining questions.

2. Are there straightforward translations of multiple comparison adjustment procedures from hypothesis testing, to interval estimation? For example, would adjustments focus on changing the $$\text{CI-level}$$ term in the confidence interval: $$\text{CI}_{\theta} = (\hat{\theta} \pm t_{(1-\text{CI-level)/2}}\hat{\sigma}_{\theta})$$?

3. How would we address step-up or step-down control procedures for CIs? Some family-wise error rate adjustments from the hypothesis testing approach to inference are 'static' in that precisely the same adjustment is made to each separate inference. For example, the Bonferroni adjustment is made by altering rejection criterion from:

• reject if $$p\le \frac{\alpha}{2}$$ to:
• reject if $$p\le \frac{\frac{\alpha}{2}}{m}$$,

but the Holm-Bonferroni step-up adjustment is not 'static', but rather made by:

• first ordering $$p$$-values smallest to largest, and then
• reject if $$p\le 1 - (1- \frac{\alpha}{2})^{\frac{1}{m+1-i}}$$, (where $$i$$ indexes the ordering of the $$p$$-values) until
• we fail to reject a null hypothesis, and automatically fail to reject all subsequent null hypotheses.

Because rejection/failure to reject is not happening with CIs (more formally, see the references below) does that mean that stepwise procedures don't translate (i.e. including all of the FDR methods)? I ought to caveat here that I am not asking how to translate CIs into hypothesis tests (the representatives of the 'visual hypothesis testing' literature cited below get at that non-trivial question).

4. What about any of those other intervals I mentioned parenthetically in 1?

1 Gosh, I sure hope I don't get in trouble with those rockin' the sweet, sweet Bayesian styles by using this word here. :)

References
Afshartous, D. and Preston, R. (2010). Confidence intervals for dependent data: Equating non-overlap with statistical significance. Computational Statistics & Data Analysis, 54(10):2296–2305.

Cumming, G. (2009). Inference by eye: reading the overlap of independent confidence intervals. Statistics In Medicine, 28(2):205–220.

Payton, M. E., Greenstone, M. H., and Schenker, N. (2003). Overlapping confidence intervals or standard error intervals: What do they mean in terms of statistical significance? Journal of Insect Science, 3(34):1–6.

Tryon, W. W. and Lewis, C. (2008). An inferential confidence interval method of establishing statistical equivalence that corrects Tryon’s (2001) reduction factor. Psychological Methods, 13(3):272–277.

• I don't have time to research a full answer now, so I'll answer in a comment. – Harvey Motulsky Sep 17 '14 at 4:21
• [The last comment got truncated.[I don't have time to research a full answer now, so I'll answer in a comment. 1) Yes it makes sense in the same situations as multiple comparisons for hypothesis testing makes sense. 2. Bonferroni, Tukey and Dunnet multiple comparisons can easily be adapted to making confidence intervals where the confidence level applies to the entire family. 3. As far as I can tell, there is no possibility of making confidence intervals from Holm method. 4. I don't have a clue! – Harvey Motulsky Sep 17 '14 at 4:34
• @HarveyMotulsky Great! As to your first two answers: (1) Why? (2) Simply by inverting the math from $p$-value adjustments to be $\alpha$-adjustments when calculating the critical values of a distribution with which one is constructing a CI? You could (nudge) always write (nudge) a formal answer instead of amplifying in the comments (nudgitty-nudge nudge). – Alexis Sep 19 '14 at 0:49

An excellent topic which is, sadly, not given enough attention.

When discussing multiple parameters and confidence intervals, a distinction should be made between simultaneous inference and selective inference. Ref.[2] gives an excellent demonstration of the matter.

Simultaneous confidence intervals mean that all the parameters are covered with $1-\alpha$ confidence.
Selective confidence intervals mean that a subset of selected parameters are covered.

These two concepts can be combined: Say you construct intervals only on parameters for which you rejected the null hypothesis. You are clearly dealing with selective inference. You may want to guarantee simultaneous coverage of selected parameters, or marginal coverage of selected parameters. The former would be the counterpart of FWER control, and the latter of FDR control.

Now more to the point: Not all testing procedures have their accompanying intervals. For FWER procedures and their accompanying intervals, see [3]. Sadly, this reference is a bit outdated. For the interval counterpart of BH FDR control, see [1] and an application in [4] (which also includes a brief review of the matter). Please note that this is a fresh and active research field so that you can expect more results in the near future.

[1] Benjamini, Y., and D. Yekutieli. “False Discovery Rate-Adjusted Multiple Confidence Intervals for Selected Parameters.” Journal of the American Statistical Association 100, no. 469 (2005): 71–81.

[2] Cox, D. R. “A Remark on Multiple Comparison Methods.” Technometrics 7, no. 2 (1965): 223–24.

[3] Hochberg, Y., and A. C. Tamhane. Multiple Comparison Procedures. New York, NY, USA: John Wiley & Sons, Inc., 1987.

[4] Rosenblatt, J. D., and Y. Benjamini. “Selective Correlations; Not Voodoo.” NeuroImage 103 (December 2014): 401–10.

I would never adjust confidence intervals for multiple testing. I am not a big fan of p-values, because I believe that estimating parameters is a better use of statistics than testing hypotheses which are never exactly true. However I concede that hypothesis testing has its value, in say a randomised controlled trial where at least one can argue that asymptotically, if a treatment doesn't work, the null hypothesis is true. However as I have said elsewhere [1], usually this involves having one primary outcome. However, confidence intervals, in the frequentist definition, don't involve hypotheses and so don't need adjustment for other, potentially irrelevant, comparisons. Suppose I was testing phenotypes associated with a particular gene, say height and blood pressure. I'd like to know how big the difference in height is between those with and without the gene, and how well I have estimated it. I don't see that the fact that I also measured blood pressure has anything to do with it. Where it could matter is that if these two were the only significant ones from hundreds we tested. Then it is likely that the differences are, by chance, larger than the expected counterfactual experiments where we only measured height and blood pressure, but did it hundreds of experiments. However in those circumstances, no simple adjustment would work, and better to give the unadjusted estimate but come clean as to how you got these comparisons. We have also published some reults on overlapping confidence intervals.[2]

[1] Campbell MJ and Swinscow TDV (2009) Statistics at Square One. 11th ed Oxford; BMJ Books Blackwell Publishing

[2]Julious SA, Campbell MJ, Walters SJ (2007) Predicting where future means will lie based on the results of the current trial. Contemporary Clinical Trials, 28, 352-357.

• Thank you for the thought provoking answer, Mike. Benjamini, Hochberg and Yekutieli seem to argue that comparisons are not "irrelevant," but in fact simultaneous: "Simultaneous coverage is also needed when an action is to be taken based on the value of all of the parameters. Thus comparing primary endpoints between two treatments in a clinical trial is likely to involve the inspection of all of them, whether they are significantly different or not. This is a clear situation where simultaneous coverage is needed." (Leaving aside the issue of selective presentation of only some CIs.) – Alexis May 11 '16 at 18:53
• Incidentally, given "I am not a big fan of p-values, because I believe that estimating parameters is a better use of statistics than testing hypotheses which are never exactly true," you might enjoy Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples?. Cheers. – Alexis May 11 '16 at 18:55
• While I agree with you that confidence intervals for parameters are superior to p-values for most forms of inference, I'm not sure if that necessarily implies that no correction for multiple comparisons is necessary with confidence intervals. Most confidence intervals are defined by the use of alpha, to specify the coverage. Even divorced from the strict hypothesis testing framework, it seems to me (naively, without bothering to do simulations to check) that it might be misleading to stick dogmatically to the nominal coverage (e.g. 95%, so alpha =0.05) when multiple comparisons are involved. – Ryan Simmons Jul 15 '16 at 18:23
• Mike Campbell said that "confidence intervals, in the frequentist definition, don't involve hypotheses and so don't need adjustment for other, potentially irrelevant, comparisons." That is an odd statement. Although CIs may not reflect "hypothesis tests" per se, they do reflect statistical tests that have a certain error rate (e.g., .05), and that error rate is inflated as the number of tests increases--by exactly the same basic mathematical principle that applies to null hypothesis tests. One doesn't escape the issue of multiple comparisons by focusing on CIs instead of p-values. – Bonferroni Nov 11 '16 at 19:43