Adjusting p-values for multiple comparisons: if Bonferroni adjustment = α/??, what's my?

Question

I'm testing the effect of two different informational treatments on individuals' attitudes towards certain policies.

I know I need to adjust my p-values for multiple comparisons. Using the Bonferroni method I would have to divide my alpha (e.g. p = 0.05) by the number of comparisons I'm making i.e. α/??.

However, I'm not sure what to count in the ??

• I'm testing the effect of treatment on preferences for 8 different policies
• I have 3 treatment groups (control, treatment 1, treatment 2), and am interested in the marginal effect of treatment (e.g. treatment 1 vs control, treatment 2 vs control)
• I also include 13 (binary) covariates/control variables

I think (hope!) the value of ?? should be 8 i.e. one for each policy - but can anyone advise?

Answer 1

You should divide alpha by the number of hypothesis tests you want to perform. It sounds like you want to test whether each of the 2 treatment group's mean preference differs from the control group's mean preference for each policy. If so, that's 2*8 hypothesis tests, so you'd divide alpha by 16.

But this is a choice to make. Do you care whether the two treatment groups have different preferences? Do you want to study the covariates?

At the end of the day, it's a balancing act. On one hand, it'd be nice to test more hypotheses. On the other hand, each additional hypothesis you test will reduce your power for any particular hypothesis. You can do a power analysis to see the impact of this for a fixed sample size n.