# What is the cause of the multiple comparisons problem?

I understand the intuition behind the MCP but I'm having trouble pinpointing exactly the cause, what is it that should be avoided, or at least accounted for.

In its most blunt definition, I agree that if I take any data and apply a brute force approach to it trying every possible null hypotheses, I'll eventually find one that can be rejected with an arbitrary alfa (e.g., 5%) and declare a discovery.

But in many definitions of MCP I read something like "the more you test the more you are likely to find", and although I agree, I don't necessarily see it as a problem (or at least the root of the problem). For example, if many researchers are analyzing the same phenomenon with the same available data, each testing its own hypothesis, it's more likely that one will reach a discovery (than if it were just one researcher), does that mean that they should be applying some type of correction to their target alfa (e.g., a Bonferroni correction)? I'm assuming the answer is no, but then it doesn't become clear why should a single researcher testing many hypotheses should (again, agreeing that the testing system can be abused and there should be a correction for that).

When does this increased chance to find a discovery (reject a null hypothesis) become a problem? When thinking about the causes there are some factors that come to mind, but I'm not sure which one of them (or others not listed here) is more related to the cause of this problem:

1. Post hoc analysis: I understand that the hypotheses should be (preferably) formulated a priori, if not, I'm just looking at the data trying to guess which hypothesis I could fit under the desired alfa.

2. Reusing data: Is the problem gone if I use different data sets for each hypothesis I test? The chance of finding a discovery will still increase the more hypotheses I test (even on different data sets).

3. Independent researchers: reusing the previous example, is the MCP related to the same research team/effort? Or it applies to multiple independent researchers working on the same problem (or even on the same or similar data)?

4. Independent hypotheses: related to the previous issue, does the problem arise (or is more strongly manifested) when the hypotheses are independent? (because I'm covering more of the search space) or the main issue is trying similar hypotheses with small variations (e.g., fine-tuning a parameter)?

I could summarized the points above, in my interpretation, as (1) and (2) being forms of reducing the search space (borrowing terminology from optimization theory) where I'm making it easier to find a discovery; and (3) and (4) as using more orthogonal search methods that cover more of this search space every time they are applied (i.e., every time a hypothesis is tested). But these are just some possible causes I could come up with, to help get an answer started, there is much more I am missing I'm sure.

This question is somewhat of a follow up from a previous one that asks why is multiple comparison a problem, raising an issue similar to the distinction between the FWER and the FDR (if I understand the question correctly). In this question I don't regard that as an issue (although I would be more inclined to use FDR), both rates imply that there is a problem when analyzing more than one hypothesis (but I fail to see the distinction from the case when I analyze different unrelated problems, finding a discovery for each one of them with 5% significance, which means that when I've "solved" 100 problems rejecting null hypotheses, 5 of them -expected value- would probably be wrong). The best answer to that question implied that there was not a definite answer to it, and maybe there isn't one for this question either, but it would still be very helpful (to me at least) to elucidate as much as possible where is the cause of the MCP error coming from.

(Another answer to the same question suggested a paper that explains the benefits of the Bayesian multilevel model perspective over the classical perspective. This is another interesting approach worth investigating but the scope of this question is the classical framework.)

There are already several questions about this problem, many worth reading (e.g., 1, 2, 3, 4) which address (from different perspectives) the issues raised above, but I still feel a more unified answer (if that is even possible) is lacking, hence this question, which I hope doesn't decrease the (already problematic) SNR.

• "…the more you test the more you are likely to find" solely due to chance. FTFY. :) That is, "solely due to chance" rather than "due to a true association." – Alexis May 25 '17 at 20:34
• I agree, it applies not just to you but the others combined. However, you shouldn't let that put you off doing exploratory data analysis that can then be followed up rigorously and individually with other data obtained independently. – Robert Jones May 25 '17 at 20:38
• See ncbi.nlm.nih.gov/pmc/articles/PMC3659368 for a famous, important, dramatic example. – whuber May 25 '17 at 21:56
• What I'm noting is several instances of the word "discovery" in the question. If you re-read the question replacing each "discovery" with "false discovery," it may help you understand the nature of the problem more clearly. – rvl May 28 '17 at 16:37
• It seems that given a dataset, the smaller the dataset and the more researchers are working on it, the more likely it is that some spurious correlation will be found in the dataset due to chance. It becomes similar to a large group of people trying to "find" winning lottery ticket numbers. A hypothesis found on one dataset needs to be independently verified on another dataset to reduce the chances that the discovery was a false one; but it depends on the size of the dataset, how many researches are working on it, and how much you can trust their data hygiene processes. – rinspy Jun 13 '17 at 13:55

You seem to assume that a researcher can tell when a discovery is made. It's not the case. Even if you "find a discovery", you can never be sure that you have done so (unless you are some kind of omniscient being), because, as abashing as it sounds, what discriminates a false alarm from a discovery in science is usually some degree of human "confidence" in the analysis.

Your intuition is roughly correct, but it may help to consider how multiple comparison undermines the assumptions of the hypothesis test itself. When you conduct a classical hypothesis test you are generating a p-value, which is a measure of the evidence against the null hypothesis. The p-value is constructed in such a way that lower values constitute greater evidence against the null, and it is distributed uniformly under the null hypothesis. This is what allows you to regard the null hypothesis as being implausible for low p-values (relative to the significance level).

Suppose you decide to test $N > 1$ hypotheses without making any adjustment to your testing method to account for multiple comparisons. Each p-value for these tests is a random variable that is uniform under the null hypothesis for that test. So if none of the alternative hypotheses of these tests are true (i.e., all the null hypotheses are true) you have $p_1, ..., p_N \sim \text{U}(0, 1)$ (these values are generally not independent). Suppose you choose a significance level $0 < \alpha < 1$ and you test all these hypotheses against that level. To do this, you look at the ordered p-values and observe that you have $p_{(1)} < ... < p_{(k)} < \alpha < p_{(k+1)} ... < p_{(N)}$ for some $0 \leqslant k \leqslant N$. This tells you that for the first $k$ tests (corresponding to the ordered p-values) you should reject the null hypothesis for each of those tests.

What is the problem here? Well, the problem is that although the p-values of each of the tests are uniform under their respective null hypotheses, the ordered p-values are not uniform. By picking out the lowest $k$ p-values that are below the significance level, you are no longer looking at random variables that are uniform under their respective null hypotheses. In fact, for large $N$, the lowest p-values are likely to have a distribution that is heavily concentrated near zero, and so these are highly likely to be below your significance level, even though (by assumption) all the null hypotheses for your tests are true.

This phenomenon occurs regardless of whether the p-values are independent or not, and therefore occurs regardless of whether you use the same data or different data to test these hypotheses. The problem of multiple comparisons is that the lower p-values of the $N$ tests will have marginal null distributions that are not uniform. Adjustments such as the Bonferroni correction attempt to deal with this by either adjusting the p-values or significance levels to create a comparison that accounts for this phenomenon.

• So, if we take the example given in the OP of one researcher performing multiple tests on one dataset vs many individual researchers performing one test each on the same dataset such that the set of p-values for the former is the same as the combination of the individual p-values for the latter, then what? The same p-value for one of the tests is significant in the latter case, but not significant after the adjustment for MCP in the former? So, when doing multiple test it is better to write a collaborative paper involving as many researcher as there are planned tests? :) – Confounded Oct 24 '18 at 9:11
• Regardless of whether you write one paper about 10 tests or ten papers about 1 test, the issue is the same --- when you look at multiple comparisons, and pick the tests with low p-values, then conditional on that choice the p-values are no longer uniform. If ten researchers write ten individual papers reporting individual test results, and you pull out the one with the lowest p-value (e.g., for a presentation), because it has the lowest p-value, then conditional on that choice the p-value is no longer uniform. – Reinstate Monica Oct 24 '18 at 9:34
• Sorry, but I am still not sure I follow the argument. Say, the same dataset is tested on being generated from 10 different distributions. And say that for 3 of these tests the p value is below some alpha threshold. So, when these tests are performed separately by individual researchers, than the ones who tested against these 3 distributions can reject the null of data coming from the particular distribution he/she tested, but if one researcher performs the tests then he cannot reject the 3 null hypothesis? – Confounded Oct 24 '18 at 9:47
• It may well be that each individual researcher (having no knowledge of the other tests) does a hypothesis test without any adjustment, against a standard significance level. However, if a person comes along and reads all those papers then they need to take account of the aggregate evidence from all of them. That means that if they cherry-pick the paper with the lowest p-value, they should not evaluate that p-value in isolation from the others. To do so would bias them towards acceptance of a false alternative-hypothesis. – Reinstate Monica Oct 24 '18 at 9:52
• (This is really part of a broader statistical issue: If the object of inference that you use is affected by your data, then the proper use of that object of inference should take account of its dependence on the data.) – Reinstate Monica Oct 24 '18 at 9:53