What exactly is multiple testing / Bonferroni Correction? Can I use it with a single AB test resulting in a p-value? I'm having trouble drawing the link between a single hypothesis test and when you would apply the Bonferroni Correction to apply for multiple testing. The way I see it, you effectively just arbitrarily choose a scaling factor for your desired alpha (signifiance) level in order to reduce the chance of false positives all around. I'm struggling to see the actual statistical interpretation behind this.
 A: When there are multiple tests, the chance of any single false positive increases. If there are $m$ simultaneous tests, the Bonferroni corrected equivalent significance cutoff would be $\alpha/m$. Given $5$ simultaneous tests, if we use $\alpha = 0.05/5 = 0.01$ then
$$
\begin{align}
P(\text{at least one significant result}) &= 1 - P(\text{no significant results})\\
&= 1 - (1-0.01)^5\\
&= 0.04900995
\end{align}
$$
Which approximately gives us back our desired 0.05 probability of a false positive. More formally, if we have $m$ tests with p-values $p_1, ... , p_m$
$$
\begin{align}
P \Bigg[{ \bigcup_{i=1}^{m}\Big( p_i<\frac{\alpha}{m}\Big) }\Bigg] &\le \sum_{i=1}^{m}P\bigg[p_i<\frac{\alpha}{m}\bigg] (\text{Boole's Inequality})\\
&\le \sum_{i=1}^{m}\frac{\alpha}{m}\\
&= m \frac{\alpha}{m}\\
&= \alpha
\end{align}
$$
A: When running a single hypothesis test at a particular significance level, the significance level of choice, $\alpha$, is the probability of incorrectly rejecting a true null hypothesis.  However the false positive rate rises as more hypothesis tests are run.  If you run $n$ hypothesis tests, a Bonferroni correction divides the significance level by $n$ to keep the probability of a false positive the same.
For example, if we use the standard rejection criteria of $0.05$, we have a $5\%$ chance of erroneously rejecting a true null hypothesis.  However as we run more and more tests, it becomes more and more probable that we will reject a null hypothesis by pure chance.  If we run, let's say, 4 tests, setting the rejection threshold to $0.05/4$ ensures that we still keep an approximate $5\%$ probability of a false positive.  However for large $n$, a Bonferroni correction becomes too conservative and there are other criteria, such as the False Discovery Rate (FDR), that are used instead.
