I understand why corrections for multiple comparisons (e.g., the Bonferroni correction) are used in some cases. For example, if I run a a single experiment and check significance of many factors, the probability that at least one of them is significant under the null hypothesis increases. It makes sense to apply a correction to avoid false positives.
My question is: suppose I run two completely different and independent experiments. For example:
- flipping a coin and checking if it's biased based on the number of heads, and
- doing an A/B test comparing the conversion rates for two different button colours.
I plan these two experiments and know that, because I'm making multiple comparisons, the probability that at least one of them yields statistical significance under the null hypothesis has increased. Specifically, using $0.05$ as the significance level, the probability of at least one significant result is $1 - 0.95^2$.
Should I apply a correction for multiple comparisons? Why or why not?
If I should, doesn't this make significance impossible in the limit as I consider all experiments?
If not, what justifies the corrections in other cases?
In the above, I am not attempting to make an absurd argument. I actually do not know if corrections are justified in this case on either a philosophical or theoretical level.