6
$\begingroup$

I want to run an experiment with 4 variants (3 and a control), comparing the 3 variants to the control. I want to correct the alpha for multiple comparisons. I have 2 ways:

  • Benjamini-Hochberg procedure
  • Bonferroni correction

What are the trade-offs of using any of the two?

$\endgroup$
1
  • 2
    $\begingroup$ For comparing three treatments to one control, check out the Dunnetts test, which is designed for exactly that scenario. $\endgroup$ Nov 2, 2023 at 17:17

1 Answer 1

10
$\begingroup$

These procedures address multiplicity in very different ways.

Bonferroni but also others like Holm's step-down, Hochberg's step-up, and Hommel's step-up will (in order from least to most powerful) control the family-wise error rate, i.e. they will ensure that the type I error rate is at most $\alpha$ across all $n$ tests, so you'll just have a single type I error with probability $\alpha$. This is appropriate in a confirmatory setting. Bonferroni is the easiest to implement but also the most conservative, it's also important to note that Hochberg and Hommel are only valid under at most weakly positive (and ideally no) correlation of your tests.

Benjamini-Hochberg (and Benjamini–Yekutieli, Storey-Tibshirani) will control the false discovery rate, i.e. they ensure that out of $n$ tests at most a fraction $\alpha$ are a type I error, not just a single one. This is usually done in more exploratory settings where possible promising findings are to be replicated. Benjamini-Hochberg in particular also assumes at most very weak dependence of tests, this assumption is relaxed in the other FDR methods. An issue here is that they don't work well if you only have a small number of tests (and I would call 4 small in this context), this has to do with having to consider division by 0 as 0 in the martingale theory that underpins BH - I'm not too familiar with the details though. The key point is that this is a minor concern when $n$ is large, but not so when $n$ is small making FDR a completely meaningless concept for $n=1$.

In your situation I would recommend a FWER procedure.

$\endgroup$
1
  • 1
    $\begingroup$ Small addition: Every procedure that controls FWER automatically also controls FDR. See Goemann & Solari (2014): 10.1002/sim.6082 for the details. $\endgroup$ Nov 2, 2023 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.