Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

It seems the Bonferroni method (dividing experimentwise alpha by # of comparisons) for choosing the p level to fix the experimentwise alpha (when doing many pairwise comparisons) is more conservative than just solving $1 - (1 - p)^k = .05$ to get the alpha to use for each of the $k$ pairwise comparisons. Why not just solve the equation?

share|improve this question
Also note that Bonferroni $\beta$ approaches Šidák one quite fast for large $k$ and small $p$ -- for $p$=1% both methods produce practically the same value. – mbq♦ May 14 '12 at 19:03
@mbq, how did you get the accents on Sidak? Is that because you have a special keyboard? I didn't find any $\LaTeX$ when I right-clicked. – gung May 14 '12 at 21:29
@gung Ctrl-C Ctrl-V -- the rest is an Unicode magic. You can also use appropriate keyboard layout or just some app showing special character palettes. – mbq♦ May 15 '12 at 11:55

2 Answers

up vote 6 down vote accepted

The Dunn-Sidak approach (your second equation) is only valid if your contrasts are orthogonal, whereas the Bonferroni adjustment is guaranteed to always hold your experimentwise alpha below your chosen value. In addition, with only a few contrasts, both methods will yield almost identical values. When combined with the fact that the Bonferroni adjustment is easier for many people to understand, remember, and calculate, those are the basic reasons for its prevalence.

share|improve this answer
What exactly does orthogonal mean in this context? – Alexander May 14 '12 at 18:41
Adding to @gung's excellent answer. With k = 10, Dunn-Sidak, by my calculations, gives 0.0052 and Bonferroni gives .005. – Peter Flom May 14 '12 at 18:43
@Alexander, try this; if that's not what you're asking, ping me again with more info. – gung May 14 '12 at 18:50
The linked Wikipedia page confuses "orthogonal" and "independent": "Because orthogonal contrasts test different aspects of the data, they are independent" (Wikipedia). However, orthogonality is a relationship between vectors, e.g., coefficient vectors of contrasts (-> dot product = 0). Independence is a relationship between random variables, e.g., test statistics. Having orthogonal contrasts does not automatically imply independent test statistics for those contrasts, since the test statistics share the same error estimate in the denominator. – caracal May 14 '12 at 19:42
@caracal, thanks for clearing that up. The part I care about is the dot product of the contrast weights equaling 0, but it's a bit to explain what's going on there & why it's important, so I figured the Wikipedia page would do. I didn't edit it, though (or perhaps read it thoroughly enough). – gung May 14 '12 at 19:49
up vote 0 down vote

Gung gave an excellent answer. i just want to add that nothing is simpler than dividing alpha by n (the number of tests). Although Bonferroni is conservative it is easy to use and does not require added assumptions as Sidak and others do. Now it is bad when individual p-values are realtively large or n gets very large. But when the raw p-values are small it works well adn the level of conservatism is minimal.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.