Extent of multiple testing correction

Question

A bit of a strange question. In my fourth year biostatistics class today, we were discussing when and when not to use multiple testing correction, and the professor made an offhand comment. He asked why we don't correct for every test we've ever done since we started doing statistics, since they are all (mostly) independent and each time we observe a result we increase our probability of drawing a false positive. He laughed it off afterwards, but why do we not do this? I'm not saying that we should, because obviously it is ludicrous, but how far is too far when it comes to correcting for tests?

We'll assume alpha = 0.05 for simplicity, and say that each test A, B, and C are not under any sort of dependency and thus independent. If I sit down and test A, B, and C, be they T tests or whatever, I obviously have to adjust for multiple correction because I am taking 0.95 to the power of three, and my chances of getting a false positive sky rocket. However, if I do A, B, and C on different days, within the contexts of different procedures, and draw different results from them, how is this any different than the former situation? We are still observing the three tests, they are still independent.

What I'm trying to get at is the logical boundary where we say to stop doing multiple testing correction. Should we only do it for one family of tests, or should we do it for a whole paper, or should we do it for every single test we've ever run?I understand how to use multiple testing correction, and use FDR / Bonferonni at work all the time. This concept just kind of took my head in circles.

Thank you for your time.

Edit: There is extended discussion of this issue in a more recent question.

Answer 1

I think the answer to your question is that multiple correction depends on the context of the problem you are solving. If you first consider a priori testing and post-hoc testing then you can see where correction for multiple tests come into play.

Let’s say you formulate a single hypothesis, collect data and test the hypothesis. No need to correct in this case obviously. If you decide a priori to carry out two or more tests on the data set you may or may not correct for multiple testing. The correction may be different for each test and may be selected using your domain knowledge. On the other hand, you may simply use one of the usual correction methods. A priori tests are generally small in number. If you had a large number of hypotheses to tests you may decide on larger sample sizes, different samples etc, etc. In other words, you can design your experiment to give you the best possible chance of drawing correct conclusions from your hypotheses.

Post-hoc tests on the other hand are performed on a set of data with no particular hypothesis in mind. You are data dredging to some extent and you will certainly need to apply Bonferroni or FDR (or your own favourite) correction.

As different data sets collected over you’re your lifetime (or for a paper) are generally independent and asking different questions, there should be no need to worry about correcting for every test ever carried out. Remember that multiple corrections protect against familywise error (i.e. protection for a family of tests) rather than individual test error. If you can logically group your tests into families I think you will find suitable multiple comparisons bounds for these families.

Answer 2

You can think of the family-wise error rate (FWER; for more information, see this article). I would say if you run a single experiment to test A, B, and C, you should apply multiple-testing correction. If you run a separate experiment for each A, B, and C, then no correction will be needed.

You may be asking why we should need to control the error rate on a per-experiment basis. Here is my opinion. Imagine that some NIH or FDA type institution mandate that you correct for every test you have ever done. Consider that you run a experiment with a single test, and that is your first experiment. No adjustment will be needed here. Now consider that you run a new experiment again with a single test, but this time it is your $1,000^{th}$ experiment. Then you would have to use $\alpha$ of 0.05/1,000 = 0.00005! Who would want to run any experiments with such a low $\alpha$? So my guess is that, when Tukey proposed the experiment-wise error rate, he may have wanted to be fair to each experiment, since each experiment takes money, time, and resources.