Why aren't multiple hypothesis corrections applied to all experiments since the dawn of time?

Question

We know that we must apply Benjamini Hochberg-like corrections for multiple hypothesis testing to experiments based on a single data set, in order to control the false discovery rate, otherwise all experiments that give a positive result could be false.

But why don't we apply this same principle to all experiments since the beginning of time, regardless of where the data comes from?

After all, over half of published scientific results which are deemed "significant" are now known to be false and irreproducible, and there is no reason why this couldn't just as easily be 100%. As scientists only tend to publish positive results, we have no idea of the number of negative results, so we have no idea if what we publish are only ever false positives - positive results that have cropped up by pure random chance under the null hypothesis. Meanwhile, there is nothing to say that the maths behind multiple hypothesis testing corrections should apply only to results from the same data set, and not to results from all experimental data acquired over time.

It seems that the whole of science has become one big fishing expedition based on false or weak hypotheses, so how can we control for this?

How can we control the false discovery rate, if all we ever publish are independent results taken without applying any correction for multiple hypothesis testing over all experiments performed to date?

Is it possible to control the false discovery rate without applying some such correction?

Answer 1

This would obviously be an absolute nightmare to do in practice, but suppose it could be done: we appoint a Statistical Sultan and everyone running a hypothesis test reports their raw $p$-values to this despot. He performs some kind of global (literally) multiple comparisons correction and replies with the corrected versions.

Would this usher in a golden age of science and reason? No, probably not.

---

Let's start by considering one pair of hypotheses, as in a $t$-test. We measure some property of two groups and want to distinguish between two hypotheses about that property: $$\begin{align} H_0:& \textrm{ The groups have the same mean.} \\ H_A:& \textrm{ The groups have different means.} \end{align}$$ In a finite sample, the means are unlikely to be exactly equal even if $H_0$ really is true: measurement error and other sources of variability can push individual values around. However, the $H_0$ hypothesis is in some sense "boring", and researchers are typically concerned with avoiding a "false positive" situation wherein they claim to have found a difference between the groups where none really exists. Therefore, we only call results "significant" if they seem unlikely under the null hypothesis, and, by convention, that unlikeliness threshold is set at 5%.

This applies to a single test. Now suppose you decide to run multiple tests and are willing to accept a 5% chance of mistakenly accepting $H_0$ for each one. With enough tests, you therefore almost certainly going to start making errors, and lots of them.

The various multiple corrections approaches are intended to help you get back to a nominal error rate that you have already chosen to tolerate for individual tests. They do so in slightly different ways. Methods that control the Family-Wise Error Rate, like the Bonferroni, Sidak, and Holm procedures, say "You wanted a 5% chance of making an error on a single test, so we'll ensure that you there's no more than a 5% chance of making any errors across all of your tests." Methods that control the False Discovery Rate instead say "You are apparently okay with being wrong up to 5% of the time with a single test, so we'll ensure that no more than 5% of your 'calls' are wrong when doing multiple tests". (See the difference?)

---

Now, suppose you attempted to control the family-wise error rate of all hypothesis tests ever run. You are essentially saying that you want a <5% chance of falsely rejecting any null hypothesis, ever. This sets up an impossibly stringent threshold and inference would be effectively useless but there's an even more pressing issue: your global correction means you are testing absolutely nonsensical "compound hypotheses" like

$$\begin{align} H_1: &\textrm{Drug XYZ changes T-cell count } \wedge \\ &\textrm{Grapes grow better in some fields } \wedge&\\ &\ldots \wedge \ldots \wedge \ldots \wedge \ldots \wedge \\&\textrm{Men and women eat different amounts of ice cream} \end{align} $$

With False Discovery Rate corrections, the numerical issue isn't quite so severe, but it is still a mess philosophically. Instead, it makes sense to define a "family" of related tests, like a list of candidate genes during a genomics study, or a set of time-frequency bins during a spectral analysis. Tailoring your family to a specific question lets you actually interpret your Type I error bound in a direct way. For example, you could look at a FWER-corrected set of p-values from your own genomic data and say "There's a <5% chance that any of these genes are false positives." This is a lot better than a nebulous guarantee that covers inferences done by people you don't care about on topics you don't care about.

The flip side of this is that he appropriate choice of "family" is debatable and a bit subjective (Are all genes one family or can I just consider the kinases?) but it should be informed by your problem and I don't believe anyone has seriously advocated defining families nearly so extensively.

---

How about Bayes?

Bayesian analysis offers coherent alternative to this problem--if you're willing to move a bit away from the Frequentist Type I/Type II error framework. We start with some non-committal prior over...well...everything. Every time we learn something, that information is combined with the prior to generate a posterior distribution, which in turn becomes the prior for the next time we learn something. This gives you a coherent update rule and you could compare different hypotheses about specific things by calculating the Bayes factor between two hypotheses. You could presumably factor out large chunks of the model, which wouldn't even make this particularly onerous.

There is a persistent...meme that Bayesian methods don't require multiple comparisons corrections. Unfortunately, the posterior odds are just another test statistic for frequentists (i.e., people who care about Type I/II errors). They don't have any special properties that control these types of errors (Why would they?) Thus, you're back in intractable territory, but perhaps on slightly more principled ground.

The Bayesian counter-argument is that we should focus on what we can know now and thus these error rates aren't as important.

---

On Reproduciblity

You seem to be suggesting that improper multiple comparisons-correction is the reason behind a lot of incorrect/unreproducible results. My sense is that other factors are more likely to be an issue. An obvious one is that pressure to publish leads people to avoid experiments that really stress their hypothesis (i.e., bad experimental design).

For example, [in this experiment] (part of Amgen's (ir)reproduciblity initative 6, it turns out that the mice had mutations in genes other than the gene of interest. Andrew Gelman also likes to talk about the Garden of Forking Paths, wherein researchers choose a (reasonable) analysis plan based on the data, but might have done other analyses if the data looked different. This inflates $p$-values in a similar way to multiple comparisons, but is much harder to correct for afterward. Blatantly incorrect analysis may also play a role, but my feeling (and hope) is that that is gradually improving.

Answer 2

I think that you deliberately paint a pessimistic view of science produced by statistics. Indeed, in my opinion, statistics is not just a set of tools providing p values. There is also a state of rigour, care and alertness about some possible effects involved in the procedure of scientific induction... and while to my mind, everything you state is roughly true, here are some of my opinions on why we have some guarantees about the knowledge we produce:

• First in general, a conclusion should not be reached only under the argument of a p value lower than a given threshold.

• Second, to my knowledge arguments of the kind of "over half of published scientific results are wrong" are relevant and interesting but are computed on the basis of p values approximately equal to 0.05 (see e.g. Confusion regarding p-values and false discovery rate). For lower p values the effect is much lower than the announced one and in practice, it is not rare to obtain p values much lower than 0.05. Moreover, many times a given hypothesis is confirmed by several sub-hypotheses which again reduces the announced effects.

• Third, the question of reproducibility is genuine but is also a problem that must be dealt by the statistician by identifying and dealing with confounding effects, group designs ... and this can be done very well if it is done with expertise and rigour.

• Finally, as I understand it, an archetypal statistical study must more or less stand on the following 5 successive steps:

  Formulate one or a few hypotheses
  Design the corresponding study
  Acquire the data
  Analyse the data
  Make conclusions about the above hypotheses (and only these ones)
  

  This general guideline prevents us from fishing expeditions as a tool to produce general conclusions.

To conclude, I would say that your intent to protect us against bad scientific conclusions by over-thresholding p-values is a bit illusory. I would prefer protecting us against bad scientific conclusions by ensuring and encouraging warned and proper analyses (and I would like to think that this is a reason why so much qualified persons are here to help others on this site).

Answer 3

Is it possible to control the false discovery rate without applying some such correction?

Yes. This is what a threshold on p values does: it sets the rate of false discoveries to that threshold. In the long run, on all tests run on true null hypotheses, only $100\,a$ percent will have a p value below $a$.

Remember that (frequentist) error rates do not concern at all any probabilities about a hypothesis tested by any individual test, but as methods for conducting tests with guaranteed long term failure rates. Correction for multiple comparisons is another method for guaranteeing long term failure rates: one for constructing compound methods which contain multiple tests so that some guaranteed long term failure rate for the compound holds.

If you conduct a single experiment with 100 tests and report that 5 of them spoke against the null, thus claiming that you have observed some real result, nobody will be impressed, knowing that on average, amongst 100 tests of true nulls, 5% will reject; the method you have employed, "conduct 100 tests and report if any of them meets the 5% threshold", has a higher failure rate than 5%. Thus, you may chose to control for multiple comparisons and report that e.g. 2 out of 100 tests had p values lower than (5 / 100 == 0.05) %. Now you employ a method which has again a guaranteed failure rate (for the error of reporting at least one significant test even though no hypotheses are false) of 5%.

That is, FWE/Family-wise error correction controls the global error rate; that if you do n tests, only $a$ times will you find a significant effect if all your tests are of true hypotheses. That is, if literally every single hypothesis anyone ever bothered to test was true, and everyone always did >1 tests per study, and everyone applied study-wise FWE control, 5% of studies would report significant results (and 5% of all individual tests would be significant under their original, uncorrected thresholds). In contrast, if everyone always tested 100 true hypotheses per study and did not apply FEW, the number of experiments reporting significant effects would exceed the guaranteed error rate of 5%. (Contrast with FDR/False Detection Rate, which is not a method that guarantees the rate of reporting any significant test in a study of multiple tests of true hypotheses.)