# Alpha adjustment for multiple testing

I understand the logic of alpha adjustment for multiple testing. However, I am confused as to whether this correction should be applied to all tests on a dataset or only the pairwise comparison in question.

For example, I have four pairwise comparisons (male vesus female, married versus other, English versus other languages, young versus old). They are from the same sample (dataset).

If I use 0.05 and Bonferroni, should my corrected alpha be 0.025 (i.e. calculated for each pairwise comparison) or 0.0125 (i.e. calculated for four pairwise comparisons in total). How does the concept of familywise error fit in here?

I must add that my interest is in each pairwise comparison (e.g. male compared to female) and NOT across the different pairs (e.g. married male versus married female).

-

On a different note, from your description it seems to me that your situation would best be analyzed by using a factorial ANOVA, with sex as factor 1, marital status as factor 2, language as factor 3, and age as factor 4. From the description (and I recognize that it is sparse) I don't see why a cell means approach (i.e., one-way ANOVA) is preferable. If you have no interest in interactions, the main effects from the factorial ANOVA are already orthogonal (at least if the $n$s are the same), and Bonferroni corrections are not relevant. Certainly it would still be possible to have more than 5% type I errors, but I'm a big believer in @John's fourth paragraph; when I'm testing theoretically suggested, a-priori, orthogonal contrasts, I don't use alpha-adjustments.

-

Your second number is the correct one. I'm not even sure how you calculate the first one. A pairwise comparison counts as 1. How are you dividing by 2?

Regardless, if you don't take 4 as the number of comparisons you're missing how the Bonferroni works. Each time you make a test there's a chance, at the level of alpha, that you make an error saying there's a real difference when there isn't one. By adjusting alpha down you make up for the fact that that chance is inflated across all of your tests, in your case 1-(1-alpha)^4, or 0.185. That's a better than 1/6 chance of seeing a significant effect by chance. For the Bonferroni adjusted alpha the chance across all 4 tests, using the formula above, is still approximately 0.05.

There are two further things to keep in mind.

Bonferroni is really an adjustment for a fishing expedition. If you have very good reasons to do these separate tests beforehand then don't worry about the correction so much. It's still true that statistically you can increase the odds of the error but that's also true across experiments you do and across years you're a researcher. I'm generally against them.

The other thing to keep in mind is that an alpha cutoff of 0.05 is generally a pretty liberal thing and is bound to have lots of Type I errors anyway. So, if you're asking here to see if you can pick a higher alpha so one of your tests squeaks through then start thinking about the real magnitude of your effect, your confidence intervals, the quality of the data you have, etc. Trying to eke out a significant difference of a test at 0.05 is almost invariably the wrong way to be thinking about things.

-
alpha correction definitely makes the list for "ritualistic substitutes for thought" in statistics. If good theory & hypotheses are guiding one's statistical testing, then alpha correction is unnecessary for reasons @John says. Without theory & hypotheses -- if one is just fishing in the data -- alpha correction doesn't help to make the inferences one is drawing any less fishy (a point that is consistently ignored in the booming fMRI-study industry: prefrontal.org/files/posters/Bennett-Salmon-2009.pdf) –  dmk38 Jan 1 '12 at 16:34
Yes, FMRI is probably the best example of a fishing expedition and the perfect place to use multiple comparisons. –  John Jan 1 '12 at 17:20
Thanks. I was taking pairwise comparison as two tests i.e. one for male and one for female. (I guess, I was focusing on the word 'pair'!). I now understand it is wrong. Thanks for pointing this out. –  Adhesh Josh Jan 2 '12 at 23:26