Alpha adjustment for multiple testing

Question

I understand the logic of alpha adjustment for multiple testing. However, I am confused as to whether this correction should be applied to all tests on a dataset or only the pairwise comparison in question.

For example, I have four pairwise comparisons (male vesus female, married versus other, English versus other languages, young versus old). They are from the same sample (dataset).

If I use 0.05 and Bonferroni, should my corrected alpha be 0.025 (i.e. calculated for each pairwise comparison) or 0.0125 (i.e. calculated for four pairwise comparisons in total). How does the concept of familywise error fit in here?

I must add that my interest is in each pairwise comparison (e.g. male compared to female) and NOT across the different pairs (e.g. married male versus married female).

Answer 1

@John has a nice answer. I particularly like the discussion about fishing expeditions and how alpha-adjustment may not be necessary. I want to add one additional aspect to this discussion. With hypothesis testing, there are two different kinds of errors to worry about: type I and type II (also called alpha error and beta error). Both kinds are bad, and we want to avoid both of them. When people talk about alpha-adjustment, they are focusing only on the possibility of type I errors (that is, saying there is a difference when there isn't one). However, adjusting alpha to minimize type I errors necessarily decreases power. Thus, it necessarily increases the probability of type II errors (that is, saying there isn't a difference when in fact there is). In addition, it's worth noting that a-priori there is no reason to believe that type I errors are worse than type II errors (despite the fact that everyone seems to assume that this must be so). Rather, which is worse will vary from situation to situation and is a judgment that must be made by the researcher. In other words, deciding on a strategy for testing multiple comparisons (e.g., an alpha-adjustment strategy) one must consider the effect of the strategy on both type I and type II errors and balance these effects relative to: the severity of these errors, how much data you have, and the cost of gathering more.

On a different note, from your description it seems to me that your situation would best be analyzed by using a factorial ANOVA, with sex as factor 1, marital status as factor 2, language as factor 3, and age as factor 4. From the description (and I recognize that it is sparse) I don't see why a cell means approach (i.e., one-way ANOVA) is preferable. If you have no interest in interactions, the main effects from the factorial ANOVA are already orthogonal (at least if the $n$s are the same), and Bonferroni corrections are not relevant. Certainly it would still be possible to have more than 5% type I errors, but I'm a big believer in @John's fourth paragraph; when I'm testing theoretically suggested, a-priori, orthogonal contrasts, I don't use alpha-adjustments.

Answer 2

Your second number is the correct one. I'm not even sure how you calculate the first one. A pairwise comparison counts as 1. How are you dividing by 2?

Regardless, if you don't take 4 as the number of comparisons you're missing how the Bonferroni works. Each time you make a test there's a chance, at the level of alpha, that you make an error saying there's a real difference when there isn't one. By adjusting alpha down you make up for the fact that that chance is inflated across all of your tests, in your case 1-(1-alpha)^4, or 0.185. That's a better than 1/6 chance of seeing a significant effect by chance. For the Bonferroni adjusted alpha the chance across all 4 tests, using the formula above, is still approximately 0.05.

There are two further things to keep in mind.

Bonferroni is really an adjustment for a fishing expedition. If you have very good reasons to do these separate tests beforehand then don't worry about the correction so much. It's still true that statistically you can increase the odds of the error but that's also true across experiments you do and across years you're a researcher. I'm generally against them.

The other thing to keep in mind is that an alpha cutoff of 0.05 is generally a pretty liberal thing and is bound to have lots of Type I errors anyway. So, if you're asking here to see if you can pick a higher alpha so one of your tests squeaks through then start thinking about the real magnitude of your effect, your confidence intervals, the quality of the data you have, etc. Trying to eke out a significant difference of a test at 0.05 is almost invariably the wrong way to be thinking about things.

Answer 3

I also like @john 's answer, but I'd add that, rather than significance, you should be more concerned with effect size, especially if you are not doing a fishing expedition. You should also be concerned about other things. Robert Abelson, in his marvelous book: Statistics as Principled Argument says that we should evaluate statistical findings based on the "MAGIC" criteria: Magnitude - how big is the effect? Articulation - Does it need a lot of qualifications and exceptions? Generality - Does it apply to a large area (e.g. lots of types of people, or whatever) Interestingness - interesting effects are better! and Credibility - Can people believe it?

I review the book