1
$\begingroup$

I have a number of data points (~10,000) compared against different confidence intervals to determine whether or not they are significantly elevated or not.

Now I would like to correct for multiple comparison testing, but the only way I know how to do this is by using the p-values from each comparison, which I don't have because they were compared to a confidence interval.

How can I correct for multiple comparison?

$\endgroup$

1 Answer 1

1
$\begingroup$

A common way to correct for $m$ multiple comparisons when it comes to hypothesis tests is the Bonferroni correction: by rejecting each null hypothesis $H_i:i\in\{1,2,...,m\}$ at a Type 1 error rate $\alpha/m$, the familywise error rate (FWER) is controlled at $\leq \alpha$.

The Bonferroni correction is derived from the Boole-Frechet inequalities, which relates the probabilities of multiple events to the probability of their intersection and the probability of their union. Given $m$ events $E_i:i\in\{1,2,...,m\}$,

$1-m+\sum^m_{i=1}P(E_i) \leq P(\bigcap^m_{i=1}E_i) \leq \min(P(E_i):i\in\{1,2,...,m\}) \leq \max(P(E_i):i\in\{1,2,...,m\}) \leq P(\bigcup^m_{i=1}E_i) \leq \sum^m_{i=1}P(E_i)$

Say the event $E_i$ is a Type 1 error for the $i$th test, then if all $P(E_i)=\alpha/m$, the FWER $P(\bigcup^m_{i=1}E_i) \leq m\alpha/m=\alpha$. It's that simple.

With multiple confidence intervals, the event $E_i$ is each interval. You're interested in controlling the probability that all confidence intervals contain their estimated parameter, say $\geq 1-\alpha$. In this case we're looking at the intersection probability, and assuming the individual confidences $P(E_i)$ are all equal: $1-\alpha = 1-m+mP(E_i) \leq P(\bigcap^m_{i=1}E_i)$. It follows that $P(E_i)=1-\alpha/m$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.