From Wikipedia's Multiple Comparison

For hypothesis testing, the problem of multiple comparisons (also known as the multiple testing problem) results from the increase in type I error that occurs when statistical tests are used repeatedly. If n independent comparisons are performed, the experiment-wide significance level $\bar{\alpha}$, also termed FWER for familywise error rate, is given by $$ \bar{\alpha} = 1-\left( 1-\alpha_\mathrm{\{per\ comparison\}} \right)^n$$

I don't understand how the comparisons can be independent? Let the multiple tests be $\{H_i, K_i, T_i, c_i), i \in I\}$, where the $i$-th test is $H_i$ versus $K_i$, with testing statistic $T_i$ and critical value $c_i$. Now given a sample $X$, the test statistics $T_i(X), i\in I$ can't be independent, and therefore the testing rules $I_{T_i(X) \geq c_i}$ can't be independent either. Am I wrong? Thanks!


Here is a very simple example. Imagine you have 4 groups, (call them groups A, B, C, and D) and you want to test whether or not the mean of each of the groups is the same. Now, you decide to perform all pairwise comparisons between the 4 groups (thus you now have to deal with multiple comparisons). Now, two of the multiple comparisons will be comparing the mean of group A versus B and the mean of group C versus D. Now, wouldn't you agree that the comparison of group A and B is independent of the comparison of group C and D? This example is somewhat trivial but still gets at the point of your question.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.