What exactly is multiple testing / Bonferroni Correction? Can I use it with a single AB test resulting in a p-value?

Question

I'm having trouble drawing the link between a single hypothesis test and when you would apply the Bonferroni Correction to apply for multiple testing. The way I see it, you effectively just arbitrarily choose a scaling factor for your desired alpha (signifiance) level in order to reduce the chance of false positives all around. I'm struggling to see the actual statistical interpretation behind this.

Answer 1

When there are multiple tests, the chance of any single false positive increases. If there are $m$ simultaneous tests, the Bonferroni corrected equivalent significance cutoff would be $\alpha/m$. Given $5$ simultaneous tests, if we use $\alpha = 0.05/5 = 0.01$ then

$$ \begin{align} P(\text{at least one significant result}) &= 1 - P(\text{no significant results})\\ &= 1 - (1-0.01)^5\\ &= 0.04900995 \end{align} $$

Which approximately gives us back our desired 0.05 probability of a false positive. More formally, if we have $m$ tests with p-values $p_1, ... , p_m$

$$ \begin{align} P \Bigg[{ \bigcup_{i=1}^{m}\Big( p_i<\frac{\alpha}{m}\Big) }\Bigg] &\le \sum_{i=1}^{m}P\bigg[p_i<\frac{\alpha}{m}\bigg] (\text{Boole's Inequality})\\ &\le \sum_{i=1}^{m}\frac{\alpha}{m}\\ &= m \frac{\alpha}{m}\\ &= \alpha \end{align} $$

Answer 2

When running a single hypothesis test at a particular significance level, the significance level of choice, $\alpha$, is the probability of incorrectly rejecting a true null hypothesis. However the false positive rate rises as more hypothesis tests are run. If you run $n$ hypothesis tests, a Bonferroni correction divides the significance level by $n$ to keep the probability of a false positive the same.

For example, if we use the standard rejection criteria of $0.05$, we have a $5\%$ chance of erroneously rejecting a true null hypothesis. However as we run more and more tests, it becomes more and more probable that we will reject a null hypothesis by pure chance. If we run, let's say, 4 tests, setting the rejection threshold to $0.05/4$ ensures that we still keep an approximate $5\%$ probability of a false positive. However for large $n$, a Bonferroni correction becomes too conservative and there are other criteria, such as the False Discovery Rate (FDR), that are used instead.