# Least conservative multiple testing correction

This is a general question regarding multiple testing when making confidence intervals. Suppose I have an unknown vector $$\mu = (\mu_1, \dots, \mu_p).$$

My goal is to estimate this vector, and create a $100(1-\alpha)\%$ confidence interval for each component $\mu_i$. I can of course make a $p$-dimensional ellipsoidal confidence region, but let's assume I don't want to do that.

Due to the problem of multiple intervals, my realized coverage probability is going to be smaller that $(1-\alpha)$, and so there is a need for correcting for this. The most common way is to use a Bonferroni correction, but Bonferroni is ridiculously conservative as $p$ increases.

My question: what is the least conservative method for correcting for multiple intervals?

--EDIT--

I am not assuming any model setup. Lets assume $\mu$ is the mean vector of a $p$ dimensional distribution $F$, which I sample from using, say Monte Carlo.

• I think it is a good question. At the same time, the answer depends on the specifics of your analysis: (1) do you use ANOVA? Linear regression?; (2) do you have/assume knowledge of the distribution underlying your stochastic component? (E.g., are your errors in a linear regression normal?) Commented Mar 7, 2016 at 20:28
• If you are not using any "model setup" then how are you generating p-values?
– DWin
Commented Mar 7, 2016 at 21:09
• Like I said, I am not testing a hypothesis, but only making confidence regions. Once I have realizations from $F$, I am calculating the sample mean, and appealing to the CLT to make large sample confidence intervals. Commented Mar 7, 2016 at 21:11
• Scheffe's method is one possibility: en.wikipedia.org/wiki/Scheff%C3%A9%27s_method Commented Mar 7, 2016 at 21:21
• To the best of my knowledge, Scheffe's is more conservative than Bonferroni if less than or equal to $p$ confidence regions are being made. This makes sense, since Scheffe's adjusts for all possible contrasts. Commented Mar 7, 2016 at 21:25

The least conservative multiple comparison adjustment is to not do one. In that case you are accepting a familywise error-rate that is (likely) larger than your per-comparison error-rate. How much larger depends on the correlation among your random variables. If you have perfect positive dependence between the coverage of any two intervals than your per comparison error rate is identical to your family-wise error rate:

$$P(\mu_1 \in CI_{u_1}, \mu_2 \in CI_{u_2}) = P(\mu_1 \in CI_{u_1}) = P(\mu_2 \in CI_{u_2})$$

Complete independence between the coverage of any two intervals means that a joint interval of $p$ intervals will have coverage of $(1-\alpha)^p$:

$$P(\mu_1 \in CI_{u_1}, \mu_2 \in CI_{u_2}) = P(\mu_1 \in CI_{u_1}) * P(\mu_2 \in CI_{u_2})$$

Bonferroni actually controls for perfect negative dependence between the coverage of any two intervals and so is the most strict:

$$P(\mu_1 \in CI_{u_1}, \mu_2 \in CI_{u_2}) = 1 - P(\mu_1 \notin CI_{u_1} \cup \mu_2 \notin CI_{u_2})$$

If you do want to control for multiple comparisons, you have other options beside Bonferroni and beside controlling for the familywise error rate. One option is to instead control for the false discovery rate, which controls the proportion of false discoveries (or in your case, missed individual confidence interval) among your rejected hypothesis.

However, probably the best approach is to use a model that means you don't have to worry about multiple comparisons. In the classic linear modelling or ANOVA-like models, Andrew Gelman has pointed out that using multi-level models in large part frees us from worrying about multiple comparisons. In the context of your question, if you think that your various $\mu$'s are related to each other with positive or negative dependence, then use a multivariate probability model or a multi-level model, whichever is more appropriate for your conceptual model and research question.

• This is not really satisfying since not doing a correction is not being conservative. Also, I guess I want to control for the family-wise error rate. Controlling for the false discovery rate seems like a different problem all together. Finally, like I said, I am not interested in making any model assumptions. For example, when I am doing rejection-sampling to estimate an integral. Commented Mar 8, 2016 at 7:14
• @Greenparker I'll only note that you did say "least conservative." Unfortunately, you are going to have to make some kind assumption. Bonferroni assumes perfect negative dependence. You could go very slightly less conservative and assume perfect independence which mean your individual $\alpha_i$ can be determined from a familywise $\alpha$ as $\alpha_i = 1 - \alpha^{1/p}$, but that won't buy you very much. To get less conservative that you will need to make assumptions, test those assumptions, and use a model. Commented Mar 8, 2016 at 16:59
• It seems really surprising that there are no better methods for a simple thing as estimating the mean from a sample. Commented Mar 8, 2016 at 17:22
• @Greenparker There are better methods, but you said you were not interested in them. Commented Mar 8, 2016 at 17:43
• Let me rephrase my statement then. It seems really surprising that if one wants to control for the family-wise error rate in a problem as simple as large sample mean estimation and inference, one either has to be very conservative or has to resort to making model assumptions. If the data is a consequence of Quasi-Monte Çarlo, Monte Carlo or MCMC, chances are one does not want to complicate the problem further my making model assumptions. In such cases, my intuition says there has to be something more reasonable we can do. Commented Mar 8, 2016 at 20:45