I understand why corrections for multiple comparisons (e.g., the Bonferroni correction) are used in some cases. For example, if I run a a single experiment and check significance of many factors, the probability that at least one of them is significant under the null hypothesis increases. It makes sense to apply a correction to avoid false positives.

My question is: suppose I run two completely different and independent experiments. For example:

  • flipping a coin and checking if it's biased based on the number of heads, and
  • doing an A/B test comparing the conversion rates for two different button colours.

I plan these two experiments and know that, because I'm making multiple comparisons, the probability that at least one of them yields statistical significance under the null hypothesis has increased. Specifically, using $0.05$ as the significance level, the probability of at least one significant result is $1 - 0.95^2$.

Should I apply a correction for multiple comparisons? Why or why not?

If I should, doesn't this make significance impossible in the limit as I consider all experiments?

If not, what justifies the corrections in other cases?

In the above, I am not attempting to make an absurd argument. I actually do not know if corrections are justified in this case on either a philosophical or theoretical level.


You have to figure out what it is you are really testing. The point is whether or not you are interested in the joint null hypothesis. If putting both $H_0s$ into the same joint null doesn't make sense, then you don't need to adjust for multiplicity.

E.g. if you test whether one or more of $N$ drugs can cure condition $X$ then $H_0$ is joint wrt the $N$ drugs. If there are three tests "drug 1 cures X", "the population of rats in China increased over the last year", " an average Xbox 1 gamer has more skill than an average PS4 gamer" then the joint $H_0$ can still be put together, and it's easy to compute the joint (familywise) p-value but such $H_0$ is meaningless.

  • $\begingroup$ How does one judge if they belong in the same joint null? $\endgroup$
    – fraser
    Sep 17 '14 at 18:23
  • $\begingroup$ By applying common sense, as I just illustrated. $\endgroup$
    – James
    Dec 3 '14 at 14:44
  • $\begingroup$ This question asked if a a correction for multiple comparisons should be applied in the described scenario. It seems you are saying "no, because doing so would violate common sense"; I agree with this, but it does not provide any insight into the philosophical or theoretical justifications. If there are no higher justifications to appeal to, that's fine - I am just curious. $\endgroup$
    – fraser
    Dec 13 '14 at 18:54

The idea behind the sharp cutoffs for the p value in the null hypothesis significance testing framework is: If a researchers, for all his experiments in his lifetime, always accepts only results with p <= 0.05 as statistically significant, he will, averaged over his career, commit a Type I error (falsely rejecting a true null hypothesis) only in 5% of his experiments.

I'd say you are justified to use a significance level of alpha=0.05 for each of two independent experiments with planned comparisons.

The consensus seems to be to keep the overall type I error rate of all comparisons per experiment to a fixed level, e.g. 0.05. This is arbitrary, but it helps to avoid the worst consequences of post-hoc comparisons and exploratory data analysis.

  • $\begingroup$ How does one tell if we are talking about 2 separate experiments vs 2 aspects of the same experiment? Is it based on independence or some other property, or is it arbitrary / best judgment? e.g., with exploratory data analysis - at one point does one consider it to be a new experiment? $\endgroup$
    – fraser
    Sep 17 '14 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.