Should corrections for multiple comparisons be used when evaluating independent experiments?

Question

I understand why corrections for multiple comparisons (e.g., the Bonferroni correction) are used in some cases. For example, if I run a a single experiment and check significance of many factors, the probability that at least one of them is significant under the null hypothesis increases. It makes sense to apply a correction to avoid false positives.

My question is: suppose I run two completely different and independent experiments. For example:

• flipping a coin and checking if it's biased based on the number of heads, and
• doing an A/B test comparing the conversion rates for two different button colours.

I plan these two experiments and know that, because I'm making multiple comparisons, the probability that at least one of them yields statistical significance under the null hypothesis has increased. Specifically, using $0.05$ as the significance level, the probability of at least one significant result is $1 - 0.95^2$.

Should I apply a correction for multiple comparisons? Why or why not?

If I should, doesn't this make significance impossible in the limit as I consider all experiments?

If not, what justifies the corrections in other cases?

In the above, I am not attempting to make an absurd argument. I actually do not know if corrections are justified in this case on either a philosophical or theoretical level.

Answer 1

You have to figure out what it is you are really testing. The point is whether or not you are interested in the joint null hypothesis. If putting both $H_0s$ into the same joint null doesn't make sense, then you don't need to adjust for multiplicity.

E.g. if you test whether one or more of $N$ drugs can cure condition $X$ then $H_0$ is joint wrt the $N$ drugs. If there are three tests "drug 1 cures X", "the population of rats in China increased over the last year", " an average Xbox 1 gamer has more skill than an average PS4 gamer" then the joint $H_0$ can still be put together, and it's easy to compute the joint (familywise) p-value but such $H_0$ is meaningless.

Answer 2

The idea behind the sharp cutoffs for the p value in the null hypothesis significance testing framework is: If a researchers, for all his experiments in his lifetime, always accepts only results with p <= 0.05 as statistically significant, he will, averaged over his career, commit a Type I error (falsely rejecting a true null hypothesis) only in 5% of his experiments.

I'd say you are justified to use a significance level of alpha=0.05 for each of two independent experiments with planned comparisons.

The consensus seems to be to keep the overall type I error rate of all comparisons per experiment to a fixed level, e.g. 0.05. This is arbitrary, but it helps to avoid the worst consequences of post-hoc comparisons and exploratory data analysis.