# Extent of multiple testing correction

A bit of a strange question. In my fourth year biostatistics class today, we were discussing when and when not to use multiple testing correction, and the professor made an offhand comment. He asked why we don't correct for every test we've ever done since we started doing statistics, since they are all (mostly) independent and each time we observe a result we increase our probability of drawing a false positive. He laughed it off afterwards, but why do we not do this? I'm not saying that we should, because obviously it is ludicrous, but how far is too far when it comes to correcting for tests?

We'll assume alpha = 0.05 for simplicity, and say that each test A, B, and C are not under any sort of dependency and thus independent. If I sit down and test A, B, and C, be they T tests or whatever, I obviously have to adjust for multiple correction because I am taking 0.95 to the power of three, and my chances of getting a false positive sky rocket. However, if I do A, B, and C on different days, within the contexts of different procedures, and draw different results from them, how is this any different than the former situation? We are still observing the three tests, they are still independent.

What I'm trying to get at is the logical boundary where we say to stop doing multiple testing correction. Should we only do it for one family of tests, or should we do it for a whole paper, or should we do it for every single test we've ever run?I understand how to use multiple testing correction, and use FDR / Bonferonni at work all the time. This concept just kind of took my head in circles.

Edit: There is extended discussion of this issue in a more recent question.

• see my comments in another post stats.stackexchange.com/questions/117735/… – Peter Oct 7 '14 at 13:18
• @Chris C; I think your question is related to stats.stackexchange.com/questions/164181/… – user83346 Aug 2 '15 at 5:23
• Related (almost duplicate): stats.stackexchange.com/questions/206592. – amoeba Apr 11 '16 at 17:39
• @amoeba Yea, I saw that one and wondered the same thing; they are essentially the exact same question. Think anything should be done about it? – Chris C Apr 11 '16 at 17:43
• I guess if somebody voted to close that other Q when it appeared it might have been closed, but by now I think the answers there are surpassing the answers here. So I am reluctant to vote to close that one as a duplicate. We can close your Q as a duplicate of that one though, or we can try to ask mods to merge one of the Qs into another (this means that the answers will get moved to the other thread). What do you think yourself? – amoeba Apr 11 '16 at 17:48

You may be asking why we should need to control the error rate on a per-experiment basis. Here is my opinion. Imagine that some NIH or FDA type institution mandate that you correct for every test you have ever done. Consider that you run a experiment with a single test, and that is your first experiment. No adjustment will be needed here. Now consider that you run a new experiment again with a single test, but this time it is your $1,000^{th}$ experiment. Then you would have to use $\alpha$ of 0.05/1,000 = 0.00005! Who would want to run any experiments with such a low $\alpha$? So my guess is that, when Tukey proposed the experiment-wise error rate, he may have wanted to be fair to each experiment, since each experiment takes money, time, and resources.