8
$\begingroup$

I am learning multiple testing and I am curious about why it matters?

I understand the mathematics behind the multiple testing problems, for example, I understand things like $\text{FWER} \le m \alpha$, where $\text{FWER}$ refers to family-wise error rates, $m$ refers to hypotheses numbers, and $\alpha$ refers to significance level. However, I don't understand the statistical ideas behind the multiple testing problems and I don't know why it matters.

Let me elaborate on that. A common example about the consequence of multiple testing is that consider a study where we have $10,000$ hypotheses, we test each of them separately on a $0.05$ significance level and reject all of the null hypotheses which we reject in its own test. Even all of the $10,000$ null hypotheses are correct, we will reject $500$ hypotheses and make $500$ Type-$1$ errors in expectation ($0.05 \times 10,000$). The literature (e.g. Chapter 13, Introduction to Statistical Learning with R, Tibshirani et al.) uses this example to argue the multiple testing matters. I think the literature implicitly assumes that making $500$ Type-$1$ errors are bad to make this argument valid.

Let us consider another example. Consider $10,000$ independent researchers testing their own hypothesis and publishing the positive results. Formally, each of the researchers is totally independent from others, i.e. they work on different projects, have different hypotheses, do different experiments. Assume all researchers are honest, they propose their null hypothesis before the experiment, test their hypothesis with $0.05$ significance level, and publish their result if and only if the test rejects the null hypothesis. If all of the null hypotheses are correct, then there will be $500$ false rejections ($0.05 \times 10,000$), i.e. $500$ Type-$1$ errors, in expectation. Then we will see $500$ false positive papers.

It seems like there is no difference between the first example and the second example. They have the same assumptions (all $10,000$ null hypotheses are correct) and they have the same results (there will be $500$ Type-$1$ errors). However, I think the second example is exactly how the science community works: independent researchers do their own research and publish the positive result. It seems like nothing goes wrong in the second example. Then why the first example is bad while the second example is not bad? What is the difference between the two examples?

$\endgroup$
6
  • 2
    $\begingroup$ The second example is still bad, see the current "reproducibility crisis" in psychology and other fields. The difference is in the fact that the badness in the first example can be addressed much more easily, because the one researcher with 10,000 hypotheses can address the multiple testing problem far more easily than 10,000 researchers can coordinate their multiple testing problem. $\endgroup$ Feb 23 at 9:48
  • $\begingroup$ Can you please write out abbreviations and define notation? Terminology is not always used in a unified matter, people don't know all the same books, and you can't assume that people know what ISLR is. $\endgroup$ Feb 23 at 9:51
  • 1
    $\begingroup$ @StephanKolassa I don't think that the second example properly describes what leads to the "reproducibility crisis". For sure "hunting for significance", i.e., selective reporting and hidden multiple testing, issues with data quality and model assumptions, and even fraud play a role there. $\endgroup$ Feb 23 at 10:54
  • 1
    $\begingroup$ @ChristianHennig: I would not presume to explain the reproducibility crisis in a single comment, and I am sure that the points you raise are very impotant (cf. Gelman's "garden of forking paths"). Nevertheless, I would say that the huge number of possible hypotheses also has an impact (again Gelman, this time "researcher degrees of freedom"). It's hard to disentangle where one problem ends and the next one starts - especially since the null hypothesis is never true. Sometimes statistics frustrates me. $\endgroup$ Feb 23 at 11:32
  • 3
    $\begingroup$ @ChristianHennig The problem of 500 published studies describing a positive effect which is either not there or is substantially overstated, when the other 19500 studies are never published, is a major contributor to "reproducibility crisis" even when there is no bad practice by the researchers involved. It is "selective reporting" by the system even when it is not by the individual $\endgroup$
    – Henry
    Feb 23 at 11:40

1 Answer 1

4
$\begingroup$

This is an interesting question, and I've thought about this as well. My current thinking is this: Hypothesis testing has to be seen within a wider research context. Generally testing a hypothesis cannot ultimately settle a scientific problem of interest. For sure 5% type I error probability is worryingly high if decisions are made and scientific statements are taken for granted based on significances alone that can have serious consequences for society or be it individual patients. There are also many other issues with significance tests (such as that tests with sufficiently large samples easily turn out significant, and this is theoretically even "correct", if the null hypothesis is not precisely true but effects are so small that they don't matter, or are at least over-interpreted based on a small p-value).

Given all this, the difference between the two situations in my view is this. One should hope (and check!) that the "independent researchers" test hypotheses that are of real substantial interest, backed op by background information and thorough subject matter considerations. A hypothesis test should never be the only "information" based on which something is claimed; even using the very same data, effect sizes and potential violations of model assumptions, problems with data quality etc. should be addressed, and even then it should be clear that, say, $p=0.035$ isn't that strong an indication that anything meaningful is going on. Ultimately one can say that these specific data don't provide evidence against the null hypothesis, or they do, weaker or stronger (indeed keeping in mind that there are thousands of scientific papers with tests published every month if not week, and that a certain number of "false significances" is to be expected), with additional careful interpretation of all further results of data analysis.

In a single study in which 10,000 hypotheses are tested, chances are that other than running the tests there isn't much further background and detailed analysis for every single test. Also chances are that significant or "most significant" results will be selectively reported, which means that the probability that something is reported as meaningful that in fact isn't is far higher than the significance level. So I indeed believe that multiple testing adjustments are more appropriate in this situation than in a situation where more thorough analysis is done and more information is taken into account. Apart from this, one can of course generally discuss the pros and cons of significance testing in all of these situations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.