Should we favor family-wise error rate control or statistical power?

Question

To counteract the multiple comparison problem, people generally use procedures that ensure strong level alpha FWER control (e.g., Bonferroni, Holm). However, these procedures necessarilly produce a loss of statistical power (increase in type II error). I am wondering why strong level alpha FWER control has been become standard pratice, given the loss of statistical power. In other words, why do we care more about type I than type II errors? What is the rationale behind this?

Answer 1

Type I errors (false positives) should be protected against where they are more costly than type II errors (false negatives). There are many context-dependent factors that need to go into considerations of those costs, but context is too often ignored by those demanding 'corrections' for multiple comparisons. Further, strict accounting of error rates requires that we ignore many aspects of evidence in favour of preserving a theoretical long run property of a family of statistical methods. Statistical errors of the first and second kind are not always (not often, in my opinion) the best lens through which to view the results of experiments when forming scientific inferences.

There is a lot needed to flesh out this answer, and I have previously written much of that in answers to related questions on this site and, conveniently, in an open-access chapter: A Reckless Guide to P-values: Local Evidence, Global Errors The part that deals directly with multiple comparisons is section 3.2, but I recommend that you start reading at the beginning.

Citing an external source is not very good form for an answer here, so I'll include a bit more. It is useful to start with an example that provides some of the basics of context.

A scenario with context

Imagine that you are in charge of a drug discovery screen and you have a million drug candidate molecules to run through a handful of assays that are expected to help winnow out the few molecules that are likely to be good candidates for further development. Would you analyse the results in a way that protect against family-wise error from a million tests? You would then only be able to find the very strongest 'hits' (i.e. molecules that appear to have activity). You would effectively discard the rest because of the power-robbing nature of type I error protection.

No, of course you wouldn't want to do that. Instead you would likely be fairly permissive in the first round of testing to be more confident of catching most of the molecules that have some activity. All of the hits will be re-tested in the normal sequence of events, and almost always any hits that get far in the testing will be synthesised in many modified forms for testing as well. Often the molecules with modest activity initially lead to the development of good drugs. But any molecules that fail to be noticed in the first few assays will not be developed at all. In this setting a false negative error is far more costly than a false positive. The latter is only temporary whereas the former is permanent.

(Yes, I know that mass screening programs do not typically utilise Neyman–Pearsonioan hypothesis tests!)

So, what are the contextual aspects of that scenario? First, the testing is done in stages so that a false positive will be identified as such by re-testing because a false positive result is very unlikely to be followed by a another false positive in a fresh dataset. Second, it gives a circumstance where the typical preference for type I protection over type II protection is inappropriate.

Of errors and evidence for inference

Now we can consider some aspects of the statistical errors and evidence.

Type I and type II error rates are the long run accounting of all-or-none 'events'. A result that leads to the null being discarded is a type I error only if the null is true, and a result that fails to discard the null is a type II error only if that null is false. We never know if the null for a particular analysis of real-world experimental results is true or false. Never (Gilbert and Sullivan require me to add here: "...almost never!").

So how can do we tot up the number of occasions on which each type of error has occurred? We don't count them. Instead, the characteristics of a hypothesis testing procedure are calibrated theoretically to give nominal error rates when applied appropriately. It is useful to think of those error rates as belonging to the test procedure rather than to the experiment in question.

Those error rates are only very discontinuously related to the evidence in the results of an experiment or study. That point is easily illustrated by consideration of two notional experimental results p=0.049 and p=0.051. The strength of evidence in the data according to the statistical model that gave the p-values is pretty much identical, but the Neyman–Pearsonian outcomes with $\alpha$=0.05 would be very different. Now consider the result of a third experiment of p=$10^{-10}$. That represents evidence against the null that is very strong indeed, but gives a conclusion in the Neyman–Pearsonian accounting that is no different from the p=0.049. The all-or-none outcome from the Neyman–Pearsonian hypothesis test is only very slightly related to the evidence.

Now, imagine that we are intent on protecting against family-wise errors and we have those three results. The p=0.049 result would almost certainly become 'not significant' like the p=0.051 result. Fine, you might say, neither of them represents very strong evidence against the null. However, how does adding the third experimental result to the family alter the nature of the evidence from the first? It doesn't.

The evidential meaning of a p-value applies to the null hypothesis in question and to no other. In that sense it is 'local', in contrast to the error rates which are 'global'. Scientific considerations leading to conclusions about the nature of the real world need to be informed by the relevant evidence, and in most cases the relevant evidence is local.

The best way to deal with the elevated risk of false positive errors in studies where there are many comparisons made is to treat the results are preliminary, provisional, and exploratory. Any interesting findings need to be followed up on with subsequent experiments designed on the basis of those results. That follow-up might be done by others but, often, they should be done before a study is published. Of course, there is nothing wrong with publishing a preliminary, hypothesis generating study. Just don't pretend that firm conclusions about the real world can be formed on the basis of just a statistical analysis of preliminary data.