A bit of a strange question. In my fourth year biostatistics class today, we were discussing when and when not to use multiple testing correction, and the professor made an offhand comment. He asked why we don't correct for every test we've ever done since we started doing statistics, since they are all (mostly) independent and each time we observe a result we increase our probability of drawing a false positive. He laughed it off afterwards, but why do we not do this? I'm not saying that we should, because obviously it is ludicrous, but how far is too far when it comes to correcting for tests?
We'll assume alpha = 0.05 for simplicity, and say that each test A, B, and C are not under any sort of dependency and thus independent. If I sit down and test A, B, and C, be they T tests or whatever, I obviously have to adjust for multiple correction because I am taking 0.95 to the power of three, and my chances of getting a false positive sky rocket. However, if I do A, B, and C on different days, within the contexts of different procedures, and draw different results from them, how is this any different than the former situation? We are still observing the three tests, they are still independent.
What I'm trying to get at is the logical boundary where we say to stop doing multiple testing correction. Should we only do it for one family of tests, or should we do it for a whole paper, or should we do it for every single test we've ever run?I understand how to use multiple testing correction, and use FDR / Bonferonni at work all the time. This concept just kind of took my head in circles.
Thank you for your time.
Edit: There is extended discussion of this issue in a more recent question.