What is the cause of the multiple comparisons problem?

Question

I understand the intuition behind the MCP but I'm having trouble pinpointing exactly the cause, what is it that should be avoided, or at least accounted for.

In its most blunt definition, I agree that if I take any data and apply a brute force approach to it trying every possible null hypotheses, I'll eventually find one that can be rejected with an arbitrary alfa (e.g., 5%) and declare a discovery.

But in many definitions of MCP I read something like "the more you test the more you are likely to find", and although I agree, I don't necessarily see it as a problem (or at least the root of the problem). For example, if many researchers are analyzing the same phenomenon with the same available data, each testing its own hypothesis, it's more likely that one will reach a discovery (than if it were just one researcher), does that mean that they should be applying some type of correction to their target alfa (e.g., a Bonferroni correction)? I'm assuming the answer is no, but then it doesn't become clear why should a single researcher testing many hypotheses should (again, agreeing that the testing system can be abused and there should be a correction for that).

When does this increased chance to find a discovery (reject a null hypothesis) become a problem? When thinking about the causes there are some factors that come to mind, but I'm not sure which one of them (or others not listed here) is more related to the cause of this problem:

1. Post hoc analysis: I understand that the hypotheses should be (preferably) formulated a priori, if not, I'm just looking at the data trying to guess which hypothesis I could fit under the desired alfa.

2. Reusing data: Is the problem gone if I use different data sets for each hypothesis I test? The chance of finding a discovery will still increase the more hypotheses I test (even on different data sets).

3. Independent researchers: reusing the previous example, is the MCP related to the same research team/effort? Or it applies to multiple independent researchers working on the same problem (or even on the same or similar data)?

4. Independent hypotheses: related to the previous issue, does the problem arise (or is more strongly manifested) when the hypotheses are independent? (because I'm covering more of the search space) or the main issue is trying similar hypotheses with small variations (e.g., fine-tuning a parameter)?

I could summarized the points above, in my interpretation, as (1) and (2) being forms of reducing the search space (borrowing terminology from optimization theory) where I'm making it easier to find a discovery; and (3) and (4) as using more orthogonal search methods that cover more of this search space every time they are applied (i.e., every time a hypothesis is tested). But these are just some possible causes I could come up with, to help get an answer started, there is much more I am missing I'm sure.

This question is somewhat of a follow up from a previous one that asks why is multiple comparison a problem, raising an issue similar to the distinction between the FWER and the FDR (if I understand the question correctly). In this question I don't regard that as an issue (although I would be more inclined to use FDR), both rates imply that there is a problem when analyzing more than one hypothesis (but I fail to see the distinction from the case when I analyze different unrelated problems, finding a discovery for each one of them with 5% significance, which means that when I've "solved" 100 problems rejecting null hypotheses, 5 of them -expected value- would probably be wrong). The best answer to that question implied that there was not a definite answer to it, and maybe there isn't one for this question either, but it would still be very helpful (to me at least) to elucidate as much as possible where is the cause of the MCP error coming from.

(Another answer to the same question suggested a paper that explains the benefits of the Bayesian multilevel model perspective over the classical perspective. This is another interesting approach worth investigating but the scope of this question is the classical framework.)

There are already several questions about this problem, many worth reading (e.g., 1, 2, 3, 4) which address (from different perspectives) the issues raised above, but I still feel a more unified answer (if that is even possible) is lacking, hence this question, which I hope doesn't decrease the (already problematic) SNR.

Answer 1

You seem to assume that a researcher can tell when a discovery is made. It's not the case. Even if you "find a discovery", you can never be sure that you have done so (unless you are some kind of omniscient being), because, as abashing as it sounds, what discriminates a false alarm from a discovery in science is usually some degree of human "confidence" in the analysis.

Answer 2

Your intuition is roughly correct, but it may help to consider how multiple comparison undermines the assumptions of the hypothesis test itself. When you conduct a classical hypothesis test you are generating a p-value, which is a measure of the evidence against the null hypothesis. The p-value is constructed in such a way that lower values constitute greater evidence against the null, and it is distributed uniformly under the null hypothesis. This is what allows you to regard the null hypothesis as being implausible for low p-values (relative to the significance level).

Suppose you decide to test $N > 1$ hypotheses without making any adjustment to your testing method to account for multiple comparisons. Each p-value for these tests is a random variable that is uniform under the null hypothesis for that test. So if none of the alternative hypotheses of these tests are true (i.e., all the null hypotheses are true) you have $p_1, ..., p_N \sim \text{U}(0, 1)$ (these values are generally not independent). Suppose you choose a significance level $0 < \alpha < 1$ and you test all these hypotheses against that level. To do this, you look at the ordered p-values and observe that you have $p_{(1)} < ... < p_{(k)} < \alpha < p_{(k+1)} ... < p_{(N)}$ for some $0 \leqslant k \leqslant N$. This tells you that for the first $k$ tests (corresponding to the ordered p-values) you should reject the null hypothesis for each of those tests.

What is the problem here? Well, the problem is that although the p-values of each of the tests are uniform under their respective null hypotheses, the ordered p-values are not uniform. By picking out the lowest $k$ p-values that are below the significance level, you are no longer looking at random variables that are uniform under their respective null hypotheses. In fact, for large $N$, the lowest p-values are likely to have a distribution that is heavily concentrated near zero, and so these are highly likely to be below your significance level, even though (by assumption) all the null hypotheses for your tests are true.

This phenomenon occurs regardless of whether the p-values are independent or not, and therefore occurs regardless of whether you use the same data or different data to test these hypotheses. The problem of multiple comparisons is that the lower p-values of the $N$ tests will have marginal null distributions that are not uniform. Adjustments such as the Bonferroni correction attempt to deal with this by either adjusting the p-values or significance levels to create a comparison that accounts for this phenomenon.