Why multiple testing matters?

Question

I am learning multiple testing and I am curious about why it matters?

I understand the mathematics behind the multiple testing problems, for example, I understand things like $\text{FWER} \le m \alpha$, where $\text{FWER}$ refers to family-wise error rates, $m$ refers to hypotheses numbers, and $\alpha$ refers to significance level. However, I don't understand the statistical ideas behind the multiple testing problems and I don't know why it matters.

Let me elaborate on that. A common example about the consequence of multiple testing is that consider a study where we have $10,000$ hypotheses, we test each of them separately on a $0.05$ significance level and reject all of the null hypotheses which we reject in its own test. Even all of the $10,000$ null hypotheses are correct, we will reject $500$ hypotheses and make $500$ Type-$1$ errors in expectation ($0.05 \times 10,000$). The literature (e.g. Chapter 13, Introduction to Statistical Learning with R, Tibshirani et al.) uses this example to argue the multiple testing matters. I think the literature implicitly assumes that making $500$ Type-$1$ errors are bad to make this argument valid.

Let us consider another example. Consider $10,000$ independent researchers testing their own hypothesis and publishing the positive results. Formally, each of the researchers is totally independent from others, i.e. they work on different projects, have different hypotheses, do different experiments. Assume all researchers are honest, they propose their null hypothesis before the experiment, test their hypothesis with $0.05$ significance level, and publish their result if and only if the test rejects the null hypothesis. If all of the null hypotheses are correct, then there will be $500$ false rejections ($0.05 \times 10,000$), i.e. $500$ Type-$1$ errors, in expectation. Then we will see $500$ false positive papers.

It seems like there is no difference between the first example and the second example. They have the same assumptions (all $10,000$ null hypotheses are correct) and they have the same results (there will be $500$ Type-$1$ errors). However, I think the second example is exactly how the science community works: independent researchers do their own research and publish the positive result. It seems like nothing goes wrong in the second example. Then why the first example is bad while the second example is not bad? What is the difference between the two examples?

Answer 1

This is an interesting question, and I've thought about this as well. My current thinking is this: Hypothesis testing has to be seen within a wider research context. Generally testing a hypothesis cannot ultimately settle a scientific problem of interest. For sure 5% type I error probability is worryingly high if decisions are made and scientific statements are taken for granted based on significances alone that can have serious consequences for society or be it individual patients. There are also many other issues with significance tests (such as that tests with sufficiently large samples easily turn out significant, and this is theoretically even "correct", if the null hypothesis is not precisely true but effects are so small that they don't matter, or are at least over-interpreted based on a small p-value).

Given all this, the difference between the two situations in my view is this. One should hope (and check!) that the "independent researchers" test hypotheses that are of real substantial interest, backed op by background information and thorough subject matter considerations. A hypothesis test should never be the only "information" based on which something is claimed; even using the very same data, effect sizes and potential violations of model assumptions, problems with data quality etc. should be addressed, and even then it should be clear that, say, $p=0.035$ isn't that strong an indication that anything meaningful is going on. Ultimately one can say that these specific data don't provide evidence against the null hypothesis, or they do, weaker or stronger (indeed keeping in mind that there are thousands of scientific papers with tests published every month if not week, and that a certain number of "false significances" is to be expected), with additional careful interpretation of all further results of data analysis.

In a single study in which 10,000 hypotheses are tested, chances are that other than running the tests there isn't much further background and detailed analysis for every single test. Also chances are that significant or "most significant" results will be selectively reported, which means that the probability that something is reported as meaningful that in fact isn't is far higher than the significance level. So I indeed believe that multiple testing adjustments are more appropriate in this situation than in a situation where more thorough analysis is done and more information is taken into account. Apart from this, one can of course generally discuss the pros and cons of significance testing in all of these situations.