# Why multiple testing matters?

I am learning multiple testing and I am curious about why it matters?

I understand the mathematics behind the multiple testing problems, for example, I understand things like $$\text{FWER} \le m \alpha$$, where $$\text{FWER}$$ refers to family-wise error rates, $$m$$ refers to hypotheses numbers, and $$\alpha$$ refers to significance level. However, I don't understand the statistical ideas behind the multiple testing problems and I don't know why it matters.

Let me elaborate on that. A common example about the consequence of multiple testing is that consider a study where we have $$10,000$$ hypotheses, we test each of them separately on a $$0.05$$ significance level and reject all of the null hypotheses which we reject in its own test. Even all of the $$10,000$$ null hypotheses are correct, we will reject $$500$$ hypotheses and make $$500$$ Type-$$1$$ errors in expectation ($$0.05 \times 10,000$$). The literature (e.g. Chapter 13, Introduction to Statistical Learning with R, Tibshirani et al.) uses this example to argue the multiple testing matters. I think the literature implicitly assumes that making $$500$$ Type-$$1$$ errors are bad to make this argument valid.

Let us consider another example. Consider $$10,000$$ independent researchers testing their own hypothesis and publishing the positive results. Formally, each of the researchers is totally independent from others, i.e. they work on different projects, have different hypotheses, do different experiments. Assume all researchers are honest, they propose their null hypothesis before the experiment, test their hypothesis with $$0.05$$ significance level, and publish their result if and only if the test rejects the null hypothesis. If all of the null hypotheses are correct, then there will be $$500$$ false rejections ($$0.05 \times 10,000$$), i.e. $$500$$ Type-$$1$$ errors, in expectation. Then we will see $$500$$ false positive papers.

It seems like there is no difference between the first example and the second example. They have the same assumptions (all $$10,000$$ null hypotheses are correct) and they have the same results (there will be $$500$$ Type-$$1$$ errors). However, I think the second example is exactly how the science community works: independent researchers do their own research and publish the positive result. It seems like nothing goes wrong in the second example. Then why the first example is bad while the second example is not bad? What is the difference between the two examples?

• The second example is still bad, see the current "reproducibility crisis" in psychology and other fields. The difference is in the fact that the badness in the first example can be addressed much more easily, because the one researcher with 10,000 hypotheses can address the multiple testing problem far more easily than 10,000 researchers can coordinate their multiple testing problem. Feb 23 at 9:48
• Can you please write out abbreviations and define notation? Terminology is not always used in a unified matter, people don't know all the same books, and you can't assume that people know what ISLR is. Feb 23 at 9:51
• @StephanKolassa I don't think that the second example properly describes what leads to the "reproducibility crisis". For sure "hunting for significance", i.e., selective reporting and hidden multiple testing, issues with data quality and model assumptions, and even fraud play a role there. Feb 23 at 10:54
• @ChristianHennig: I would not presume to explain the reproducibility crisis in a single comment, and I am sure that the points you raise are very impotant (cf. Gelman's "garden of forking paths"). Nevertheless, I would say that the huge number of possible hypotheses also has an impact (again Gelman, this time "researcher degrees of freedom"). It's hard to disentangle where one problem ends and the next one starts - especially since the null hypothesis is never true. Sometimes statistics frustrates me. Feb 23 at 11:32
• @ChristianHennig The problem of 500 published studies describing a positive effect which is either not there or is substantially overstated, when the other 19500 studies are never published, is a major contributor to "reproducibility crisis" even when there is no bad practice by the researchers involved. It is "selective reporting" by the system even when it is not by the individual Feb 23 at 11:40

Given all this, the difference between the two situations in my view is this. One should hope (and check!) that the "independent researchers" test hypotheses that are of real substantial interest, backed op by background information and thorough subject matter considerations. A hypothesis test should never be the only "information" based on which something is claimed; even using the very same data, effect sizes and potential violations of model assumptions, problems with data quality etc. should be addressed, and even then it should be clear that, say, $$p=0.035$$ isn't that strong an indication that anything meaningful is going on. Ultimately one can say that these specific data don't provide evidence against the null hypothesis, or they do, weaker or stronger (indeed keeping in mind that there are thousands of scientific papers with tests published every month if not week, and that a certain number of "false significances" is to be expected), with additional careful interpretation of all further results of data analysis.