We know that we must apply Benjamini Hochberg-like corrections for multiple hypothesis testing to experiments based on a single data set, in order to control the false discovery rate, otherwise all experiments that give a positive result could be false.

But why don't we apply this same principle to all experiments since the beginning of time, regardless of where the data comes from?

After all, over half of published scientific results which are deemed "significant" are now known to be false and irreproducible, and there is no reason why this couldn't just as easily be 100%. As scientists only tend to publish positive results, we have no idea of the number of negative results, so we have no idea if what we publish are only ever false positives - positive results that have cropped up by pure random chance under the null hypothesis. Meanwhile, there is nothing to say that the maths behind multiple hypothesis testing corrections should apply only to results from the same data set, and not to results from all experimental data acquired over time.

It seems that the whole of science has become one big fishing expedition based on false or weak hypotheses, so how can we control for this?

How can we control the false discovery rate, if all we ever publish are independent results taken without applying any correction for multiple hypothesis testing over all experiments performed to date?

Is it possible to control the false discovery rate without applying some such correction?

  • 2
    $\begingroup$ Your own question meta.stats.stackexchange.com/questions/3049/… applies. This bundles several controversial (in some cases highly exaggerated) statements with several big questions. I think this goes against the consensus of advice already given to you. $\endgroup$
    – Nick Cox
    Commented Apr 11, 2016 at 7:59
  • 3
    $\begingroup$ Sorry, I don't know what responses you are referring to: I see no references here. I am not voting to close, nor (naturally) do I have any desire or power to stop people answering. But, for example, "since the dawn of time" is a small example of pointless exaggeration and there are several others in your post. Being provocative for its own sake won't, in my experience of this site, itself help your underlying question. Readers have to strip the style away from the substance. $\endgroup$
    – Nick Cox
    Commented Apr 11, 2016 at 8:11
  • 3
    $\begingroup$ Thanks for the invitation, but life is short. I'll let my cross-reference to the meta thread stand as my major point. I've stated my opinion on style and substance, which can stand or fall on its merits here. $\endgroup$
    – Nick Cox
    Commented Apr 11, 2016 at 8:23
  • 10
    $\begingroup$ If I'm doing science, I don't much care what false discoveries you made. Indeed, in respect of making a particular scientific claim, I may not much care what other false discoveries I made. If I'm not doing science, I may not even care about what other false discoveries I made in this particular analysis -- because if I choose my type I error rate based on the relative costs of the two types of errors, I've already chosen the tradeoff between the two, and shouldn't be correcting for multiple comparisons at all. $\endgroup$
    – Glen_b
    Commented Apr 11, 2016 at 11:47
  • 2
    $\begingroup$ Fortunately others have presented views similar to mine with cogency and clarity. As an extra comment, I advise against conflating science (whatever is of some lasting value) with its literature. There are many ways in which the literature disappoints: vagueness, triviality, logical errors, etc. In the abstract everyone is disconcerted by the thought of all those published false positive tests, but they have to be believed and acted upon to having lasting effects. (If it's a drug trial, that can be a big deal.) So, there are many things to worry about, but I don't think science is doomed. $\endgroup$
    – Nick Cox
    Commented Apr 12, 2016 at 8:34

3 Answers 3


This would obviously be an absolute nightmare to do in practice, but suppose it could be done: we appoint a Statistical Sultan and everyone running a hypothesis test reports their raw $p$-values to this despot. He performs some kind of global (literally) multiple comparisons correction and replies with the corrected versions.

Would this usher in a golden age of science and reason? No, probably not.

Let's start by considering one pair of hypotheses, as in a $t$-test. We measure some property of two groups and want to distinguish between two hypotheses about that property: $$\begin{align} H_0:& \textrm{ The groups have the same mean.} \\ H_A:& \textrm{ The groups have different means.} \end{align}$$ In a finite sample, the means are unlikely to be exactly equal even if $H_0$ really is true: measurement error and other sources of variability can push individual values around. However, the $H_0$ hypothesis is in some sense "boring", and researchers are typically concerned with avoiding a "false positive" situation wherein they claim to have found a difference between the groups where none really exists. Therefore, we only call results "significant" if they seem unlikely under the null hypothesis, and, by convention, that unlikeliness threshold is set at 5%.

This applies to a single test. Now suppose you decide to run multiple tests and are willing to accept a 5% chance of mistakenly accepting $H_0$ for each one. With enough tests, you therefore almost certainly going to start making errors, and lots of them.

The various multiple corrections approaches are intended to help you get back to a nominal error rate that you have already chosen to tolerate for individual tests. They do so in slightly different ways. Methods that control the Family-Wise Error Rate, like the Bonferroni, Sidak, and Holm procedures, say "You wanted a 5% chance of making an error on a single test, so we'll ensure that you there's no more than a 5% chance of making any errors across all of your tests." Methods that control the False Discovery Rate instead say "You are apparently okay with being wrong up to 5% of the time with a single test, so we'll ensure that no more than 5% of your 'calls' are wrong when doing multiple tests". (See the difference?)

Now, suppose you attempted to control the family-wise error rate of all hypothesis tests ever run. You are essentially saying that you want a <5% chance of falsely rejecting any null hypothesis, ever. This sets up an impossibly stringent threshold and inference would be effectively useless but there's an even more pressing issue: your global correction means you are testing absolutely nonsensical "compound hypotheses" like

$$\begin{align} H_1: &\textrm{Drug XYZ changes T-cell count } \wedge \\ &\textrm{Grapes grow better in some fields } \wedge&\\ &\ldots \wedge \ldots \wedge \ldots \wedge \ldots \wedge \\&\textrm{Men and women eat different amounts of ice cream} \end{align} $$

With False Discovery Rate corrections, the numerical issue isn't quite so severe, but it is still a mess philosophically. Instead, it makes sense to define a "family" of related tests, like a list of candidate genes during a genomics study, or a set of time-frequency bins during a spectral analysis. Tailoring your family to a specific question lets you actually interpret your Type I error bound in a direct way. For example, you could look at a FWER-corrected set of p-values from your own genomic data and say "There's a <5% chance that any of these genes are false positives." This is a lot better than a nebulous guarantee that covers inferences done by people you don't care about on topics you don't care about.

The flip side of this is that he appropriate choice of "family" is debatable and a bit subjective (Are all genes one family or can I just consider the kinases?) but it should be informed by your problem and I don't believe anyone has seriously advocated defining families nearly so extensively.

How about Bayes?

Bayesian analysis offers coherent alternative to this problem--if you're willing to move a bit away from the Frequentist Type I/Type II error framework. We start with some non-committal prior over...well...everything. Every time we learn something, that information is combined with the prior to generate a posterior distribution, which in turn becomes the prior for the next time we learn something. This gives you a coherent update rule and you could compare different hypotheses about specific things by calculating the Bayes factor between two hypotheses. You could presumably factor out large chunks of the model, which wouldn't even make this particularly onerous.

There is a persistent...meme that Bayesian methods don't require multiple comparisons corrections. Unfortunately, the posterior odds are just another test statistic for frequentists (i.e., people who care about Type I/II errors). They don't have any special properties that control these types of errors (Why would they?) Thus, you're back in intractable territory, but perhaps on slightly more principled ground.

The Bayesian counter-argument is that we should focus on what we can know now and thus these error rates aren't as important.

On Reproduciblity

You seem to be suggesting that improper multiple comparisons-correction is the reason behind a lot of incorrect/unreproducible results. My sense is that other factors are more likely to be an issue. An obvious one is that pressure to publish leads people to avoid experiments that really stress their hypothesis (i.e., bad experimental design).

For example, [in this experiment] (part of Amgen's (ir)reproduciblity initative 6, it turns out that the mice had mutations in genes other than the gene of interest. Andrew Gelman also likes to talk about the Garden of Forking Paths, wherein researchers choose a (reasonable) analysis plan based on the data, but might have done other analyses if the data looked different. This inflates $p$-values in a similar way to multiple comparisons, but is much harder to correct for afterward. Blatantly incorrect analysis may also play a role, but my feeling (and hope) is that that is gradually improving.

  • $\begingroup$ Thanks Matt. I love the idea of a "Statistical Sultan". Still, is it possible to control the false discovery rate without applying some such correction? $\endgroup$
    – Kelvin
    Commented Apr 11, 2016 at 9:44
  • 10
    $\begingroup$ The point I was trying to make is that it doesn't make sense to worry about The False Discovery Rate (or familywise error rate) across all human endeavors. Doing so would require so much risk aversion that you'd never get anything done. Instead, you keep the FDR/FWER for individual experiments pretty low and try to replicate the significant things that are also interesting/useful/etc. $\endgroup$ Commented Apr 11, 2016 at 9:51
  • $\begingroup$ Thanks, I guess in the end it all comes down to replication of things that matter. This is fully consistent with the philosophy of science, that no hypothesis can ever be proven, only strengthened over time by repeated experiments. $\endgroup$
    – Kelvin
    Commented Apr 11, 2016 at 9:58
  • 3
    $\begingroup$ +1 for the Statistical Sultan. One important consideration: how should Sultan handle the fact that p-values arrive successively? A lousy p=0.045 arriving first will be deemed significant but after a couple of centuries will have no chance? That does not seem to make sense (cc to @Kelvin). Another consideration: imagine that Sultan were to wait for let's say 1 year and apply the correction to all the results from this past year; I wonder what the adjusted alpha threshold would actually become in practice. Any ideas on that, Matt? That's (falsely!) assuming everybody agrees on a common alpha. $\endgroup$
    – amoeba
    Commented Apr 11, 2016 at 12:42
  • 2
    $\begingroup$ @amoeba, that's an interesting question and I'm not sure I know. Our beloved Data Despot could force everyone to use some sort of sequential design, which might help, but he's still testing this strange compound hypothesis. Alternately, we could all become Bayesians and stop worrying about our Type I/II error track record most of the time. This is a little cheap (if you can't beat 'em, ignore 'em!), but I think it's close to how people behave in practice. $\endgroup$ Commented Apr 11, 2016 at 15:38

I think that you deliberately paint a pessimistic view of science produced by statistics. Indeed, in my opinion, statistics is not just a set of tools providing p values. There is also a state of rigour, care and alertness about some possible effects involved in the procedure of scientific induction... and while to my mind, everything you state is roughly true, here are some of my opinions on why we have some guarantees about the knowledge we produce:

  • First in general, a conclusion should not be reached only under the argument of a p value lower than a given threshold.

  • Second, to my knowledge arguments of the kind of "over half of published scientific results are wrong" are relevant and interesting but are computed on the basis of p values approximately equal to 0.05 (see e.g. Confusion regarding p-values and false discovery rate). For lower p values the effect is much lower than the announced one and in practice, it is not rare to obtain p values much lower than 0.05. Moreover, many times a given hypothesis is confirmed by several sub-hypotheses which again reduces the announced effects.

  • Third, the question of reproducibility is genuine but is also a problem that must be dealt by the statistician by identifying and dealing with confounding effects, group designs ... and this can be done very well if it is done with expertise and rigour.

  • Finally, as I understand it, an archetypal statistical study must more or less stand on the following 5 successive steps:

    Formulate one or a few hypotheses
    Design the corresponding study
    Acquire the data
    Analyse the data
    Make conclusions about the above hypotheses (and only these ones)

    This general guideline prevents us from fishing expeditions as a tool to produce general conclusions.

To conclude, I would say that your intent to protect us against bad scientific conclusions by over-thresholding p-values is a bit illusory. I would prefer protecting us against bad scientific conclusions by ensuring and encouraging warned and proper analyses (and I would like to think that this is a reason why so much qualified persons are here to help others on this site).

  • 2
    $\begingroup$ I don't think it helps to get defensive. The current issue of irreproducibility in science is not just "interesting", it is at a crisis point and has been on the cover of Nature and even the Economist, since whether to believe in a particular study (or even the efficacy of an approved drug) is now no better than a coin flip, despite billions of dollars invested. $\endgroup$
    – Kelvin
    Commented Apr 11, 2016 at 8:08
  • 6
    $\begingroup$ I agree that a crisis exists. My point is that you can inspect the quality of the coin. Not all papers are of the same quality and from my experience it is sometimes easy to point paper flawed. I do not deny the problem I deny the solution : just produce proper analysis :) $\endgroup$
    – beuhbbb
    Commented Apr 11, 2016 at 8:18
  • $\begingroup$ Ok, thanks, I respect your answer. But still from a statistical point of view, and regardless of the quality of experiments, we can never control the overall false discovery rate without applying some such correction, can we? $\endgroup$
    – Kelvin
    Commented Apr 11, 2016 at 8:24

Is it possible to control the false discovery rate without applying some such correction?

Yes. This is what a threshold on p values does: it sets the rate of false discoveries to that threshold. In the long run, on all tests run on true null hypotheses, only $100\,a$ percent will have a p value below $a$.

Remember that (frequentist) error rates do not concern at all any probabilities about a hypothesis tested by any individual test, but as methods for conducting tests with guaranteed long term failure rates. Correction for multiple comparisons is another method for guaranteeing long term failure rates: one for constructing compound methods which contain multiple tests so that some guaranteed long term failure rate for the compound holds.

If you conduct a single experiment with 100 tests and report that 5 of them spoke against the null, thus claiming that you have observed some real result, nobody will be impressed, knowing that on average, amongst 100 tests of true nulls, 5% will reject; the method you have employed, "conduct 100 tests and report if any of them meets the 5% threshold", has a higher failure rate than 5%. Thus, you may chose to control for multiple comparisons and report that e.g. 2 out of 100 tests had p values lower than (5 / 100 == 0.05) %. Now you employ a method which has again a guaranteed failure rate (for the error of reporting at least one significant test even though no hypotheses are false) of 5%.

That is, FWE/Family-wise error correction controls the global error rate; that if you do n tests, only $a$ times will you find a significant effect if all your tests are of true hypotheses. That is, if literally every single hypothesis anyone ever bothered to test was true, and everyone always did >1 tests per study, and everyone applied study-wise FWE control, 5% of studies would report significant results (and 5% of all individual tests would be significant under their original, uncorrected thresholds). In contrast, if everyone always tested 100 true hypotheses per study and did not apply FEW, the number of experiments reporting significant effects would exceed the guaranteed error rate of 5%. (Contrast with FDR/False Detection Rate, which is not a method that guarantees the rate of reporting any significant test in a study of multiple tests of true hypotheses.)

  • 6
    $\begingroup$ What you call "rate of false discoveries" in your first paragraph is not what is known as "false discovery rate". $\endgroup$
    – amoeba
    Commented Apr 11, 2016 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.