1
$\begingroup$

Does a "uniformly most powerful" (UMP) test also imply that it will make the same or fewer type I errors? Specifically, is there ever a (statistical, not convenience) reason to use a simple Bonferroni multiple test correction over a Holm-Bonferroni, given that the latter is UMP?

$\endgroup$
  • 1
    $\begingroup$ Both the Bonferroni procedure and the Holm-Bonferroni procedure guarantee that the (family-wise) type I error rate is at most $\alpha$. The fact that H-B is UMP implies that the type II error rate is uniformly smaller. The only real reason I can see to use Bonferroni (over Holm-Bonferroni) is simplicity. $\endgroup$ – knrumsey Dec 11 '19 at 2:40
1
$\begingroup$

No.

It’s there in the name: POWERful. Something must be going on with the power, so it is type II error rate that improves, not type I error rate.

By choosing $\alpha$, we decide what kind of type I error rate we accept. The power then falls out of a power calculation, depending on the effect size and sample size.

What UMP means is that, given all of the factors the affect power (chosen $\alpha$, sample size, and effect size), the UMP test never has lower power than another test, and it at least sometimes has superior power.

I suppose this could be reversed to say that, for a given acceptable level of type II errors, the UMP test will be the best (or tied for best) when it comes to type I error rate, but we usually think of power as depending on the type I error rate that we set instead of the other way around (though you can do either, and that’s kind of what’s going on when machine learning people play with ROCAUC).

(My language may be a bit more evocative of two-sample t-testing than I truly intend, though I think it should be clear how to modify what I said if you want to do a KS test between two distributions or a one-sample t-test, for instance.)

Regarding Bonferroni versus Bonferroni-Holm, the reason to use Bonferroni is convenience. BH is superior, but at a cost of ease to calculate and explain. It’s easy to show the notorious jelly bean XKCD to your boss and say that you’ll divide by the number of tests in order to avoid such an erroneous conclusion.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.