What exactly is multiple testing / Bonferroni Correction? Can I use it with a single AB test resulting in a p-value?

I'm having trouble drawing the link between a single hypothesis test and when you would apply the Bonferroni Correction to apply for multiple testing. The way I see it, you effectively just arbitrarily choose a scaling factor for your desired alpha (signifiance) level in order to reduce the chance of false positives all around. I'm struggling to see the actual statistical interpretation behind this.

When there are multiple tests, the chance of any single false positive increases. If there are $m$ simultaneous tests, the Bonferroni corrected equivalent significance cutoff would be $\alpha/m$. Given $5$ simultaneous tests, if we use $\alpha = 0.05/5 = 0.01$ then

\begin{align} P(\text{at least one significant result}) &= 1 - P(\text{no significant results})\\ &= 1 - (1-0.01)^5\\ &= 0.04900995 \end{align}

Which approximately gives us back our desired 0.05 probability of a false positive. More formally, if we have $m$ tests with p-values $p_1, ... , p_m$

\begin{align} P \Bigg[{ \bigcup_{i=1}^{m}\Big( p_i<\frac{\alpha}{m}\Big) }\Bigg] &\le \sum_{i=1}^{m}P\bigg[p_i<\frac{\alpha}{m}\bigg] (\text{Boole's Inequality})\\ &\le \sum_{i=1}^{m}\frac{\alpha}{m}\\ &= m \frac{\alpha}{m}\\ &= \alpha \end{align}

When running a single hypothesis test at a particular significance level, the significance level of choice, $\alpha$, is the probability of incorrectly rejecting a true null hypothesis. However the false positive rate rises as more hypothesis tests are run. If you run $n$ hypothesis tests, a Bonferroni correction divides the significance level by $n$ to keep the probability of a false positive the same.

For example, if we use the standard rejection criteria of $0.05$, we have a $5\%$ chance of erroneously rejecting a true null hypothesis. However as we run more and more tests, it becomes more and more probable that we will reject a null hypothesis by pure chance. If we run, let's say, 4 tests, setting the rejection threshold to $0.05/4$ ensures that we still keep an approximate $5\%$ probability of a false positive. However for large $n$, a Bonferroni correction becomes too conservative and there are other criteria, such as the False Discovery Rate (FDR), that are used instead.

• I guess I'm confused as to what it means to run multiple tests. I'm running a simple AB test in a web setting. Do I need to use a multiple testing correction and/or FDR?
– John
May 29 '15 at 23:44
• Running multiple tests just means running multiple hypothesis tests. If it's one AB test with one p-value, then you don't need multiple testing. In general, if you aren't running like 1000 hypothesis tests, you don't need FDR. May 30 '15 at 1:10
• Do the multiple hypothesis tests have to be making inferences on the same populations for it to be 'multiple testing'?
– John
Jun 2 '15 at 19:44
• No, since there will still be a 0.05 probability of a Type I Error, even if you're running a bunch of tests across many populations. Jun 3 '15 at 15:39