# Confusion with false discovery rate and multiple testing (on Colquhoun 2014)

I have read this great paper by David Colquhoun: An investigation of the false discovery rate and the misinterpretation of p-values (2014). In essence, he explains why false discovery rate (FDR) can be as high as $30\%$ even though we control for type I error with $\alpha=0.05$.

However I am still confused as to what happens if I apply FDR control in the case of multiple testing.

Say, I have performed a test for each of many variables, and calculated the $q$-values using Benjamini-Hochberg procedure. I got one variable that is significant with $q=0.049$. I am asking what is the FDR for this finding?

Can I safely assume that in the long run, if I do such analysis on a regular basis, the FDR is not $30\%$, but below $5\%$, because I used Benjamini-Hochberg? That feels wrong, I would say that the $q$-value corresponds to the $p$-value in Colquhoun's paper and his reasoning applies here as well, so that by using a $q$-threshold of $0.05$ I risk to "make fool of myself" (as Colquhoun puts it) in $30\%$ of the cases. However, I tried to explain it more formally and I failed.

• Hey @January, I wonder why would you offer such a large bounty (250) and then never come back to award it and/or check the answers! Hope you are well. Apr 14, 2015 at 16:34
• Two manuscripts came down on me like a ton of bricks and I totally forgot about it. Apr 22, 2015 at 7:17

It so happens that by coincidence I read this same paper just a couple of weeks ago. Colquhoun mentions multiple comparisons (including Benjamini-Hochberg) in section 4 when posing the problem, but I find that he does not make the issue clear enough -- so I am not surprised to see your confusion.

The important point to realize is that Colquhoun is talking about the situation without any multiple comparison adjustments. One can understand Colquhoun's paper as adopting a reader's perspective: he essentially asks what false discovery rate (FDR) can he expect when he reads scientific literature, and this means what is the expected FDR when no multiple comparison adjustments were done. Multiple comparisons can be taken into account when running multiple statistical tests in one study, e.g. in one paper. But nobody ever adjusts for multiple comparisons across papers.

If you actually control FDR, e.g. by following Benjamini-Hochberg (BH) procedure, then it will be controlled. The problem is that running BH procedure separately in each study, does not guarantee overall FDR control.

Can I safely assume that in the long run, if I do such analysis on a regular basis, the FDR is not $$30\%$$, but below $$5\%$$, because I used Benjamini-Hochberg?

No. If you use BH procedure in every paper, but independently in each of your papers, then you can essentially interpret your BH-adjusted $$p$$-values as normal $$p$$-values, and what Colquhoun says still applies.

### General remarks

The answer to Colquhoun's question about the expected FDR is difficult to give because it depends on various assumptions. If e.g. all the null hypotheses are true, then FDR will be $$100\%$$ (i.e. all "significant" findings would be statistical flukes). And if all nulls are in reality false, then FDR will be zero. So the FDR depends on the proportion of true nulls, and this is something that has be externally estimated or guessed, in order to estimate the FDR. Colquhoun gives some arguments in favor of the $$30\%$$ number, but this estimate is highly sensitive to the assumptions.

I think the paper is mostly reasonable, but I dislike that it makes some claims sound way too bold. E.g. the first sentence of the abstract is:

If you use $$p=0.05$$ to suggest that you have made a discovery, you will be wrong at least $$30\%$$ of the time.

This is formulated too strongly and can actually be misleading.

• Granted, I only skimmed through the paper rather quickly, but it seems to me that he is essentially just reiterating the well-known conceit that it is easy to find spurious effects in large sample sizes (e.g. figure 1). Which isn't to say it isn't meaningful, but rather that I feel it should have a different (and less boldly stated) interpretation than the author provides. Apr 8, 2015 at 17:26
• I'm not sure why @RyanSimmons says that I was "essentially just reiterating the well-known conceit that it is easy to find spurious effects in large sample sizes". It was nothing to do with large sample sizes! I'd really welcome an explanation of why he thinks the paper should have "a different (and less boldly stated) interpretation". Apr 10, 2015 at 14:05
• "But nobody ever adjusts for multiple comparisons across papers. It would also be pretty impossible to do." I thought one of the advantages of false discovery rate adjustments over familywise error rate adjustments was that while the latter require a definition of family, the former is scalable across an arbitrary number of comparisons? Apr 12, 2015 at 23:14
• @Alexis, I looked on wikipedia and it does say that FDR control is "scalable", but I don't know what exactly that is supposed to mean (I am not an expert). However, it is easy to see that if each paper has only one test performed, then Benjamini-Hochberg procedure does exactly nothing: it rejects if $p\le \alpha$ and accepts otherwise. Repeating this in many papers is equivalent to not using any FDR control and is certainly not equivalent to first collecting all the $p$-values across the papers, and then applying Benjamini-Hochberg procedure. Apr 13, 2015 at 9:05
• Well, what you describe is certainly not a multiple comparison procedure. However, performing FDR-based adjustment methods on, say 5 tests, and then adding 20 more to that set of 10 and performing the same method again preserves the rejection probabilities under FDR, but these rejection probabilities change under FWER. Dunn's Bonferroni adjustment provides a rather dramatic example. Apr 13, 2015 at 16:32

Benjamini & Hochberg define false discovery rate in the same way that I do, as the fraction of positive tests that are false positives. So if you use their procedure for multiple comparisons you control FDR properly. It's worth noting, though, that there are quite a lot of variants on the B-H method. Benjamini's seminars at Berkeley are on Youtube, and well worth watching:

I'm not sure why @amoeba says "This is formulated too strongly and can actually be misleading". I'd be interested to know why he/she thinks that. The most persuasive argument comes from the simulated t tests (section 6). That mimics what almost everyone does in practice and it shows that if you observe P close to 0.047, and claim to have made a discovery, you'll be wrong at least 26% of the time. What can go wrong?

Of course, I should not describe this as a minimum. It's what you get if you assume that there's a 50% chance of the there being a real effect. Of course if you assume that most of your hypotheses are correct in advance, then you can get a lower FDR than 26%, but can you imagine the hilarity that would greet a claim that you'd made a discovery on the basis of the assumption that you were 90% sure in advance that your conclusion would be true. 26% is the minimum FDR given that it isn't a reasonable basis for inference to assume any prior probability greater than 0.5.

Given that hunches frequently don't stand up when tested, it could well be that there is only a 10% chance of any particular hypothesis being true, and in that case the FDR would be a disastrous 76%.

It's true that all this is contingent on the null hypothesis being that there is zero difference (the so called point null). Other choices can give different results. But the point null is what almost everyone uses in real life (though the may not be aware of it). Furthermore the point null seems to me to be entirely appropriate thing to use. It's sometimes objected that true differences are never exactly zero. I disagree. We want to tell whether are not our results are distinguishable from the case where both groups are given identical treatments, so the true difference is exactly zero. If we decide that out data are not compatible with that view, we go on to estimate the effect size. and at that point we make the separate judgment about whether the effect, though real, is big enough to be important in practice. There is some vigorous discussion of these topics on Deborah Mayo's blog.

@amoeba Thanks for you response.

What the discussion on Mayo's blog shows is mostly that Mayo doesn't agree with me, though she hasn't made clear why, to me at least). Stephen Senn points out correctly that you can get a different answer if you postulate a different prior distribution. That seems to me to be interesting only to subjective Bayesians.

It's certainly irrelevant to everyday practice which always assumes a point null. And as I explained, that seems to me to be a perfectly sensible thing to do.

Many professional statisticians have come to conclusions much the same as mine. Try Sellke & Berger, and Valen Johnson (refs in my paper). There is nothing very controversial (or very original) about my claims.

Your other point, about assuming a 0.5 prior, doesn't seem to me to be an assumption at all. As I explained above, anything above 0.5 woold be unacceptable in practice. And anything below 0.5 makes the false discovery rate even higher (eg 76% if prior is 0.1). Therefore it's perfectly reasonable to say that 26% is the minimum false discovery rate that you can expect if you observe P = 0.047 in a single experiment.

I've been thinking more about this question. My definition of FDR is the same as Benjamini's -the fraction of positive tests that are false. But it is applied to a quite different problem, the interpretation of a single test. With hindsight it might have been better if I'd picked a different term.

In the case of a single test, B&H leaves the P value unchanged, so it does not say anything about the false discovery rate in the sense that I use the term.

es of course you are right. Benjamini & Hochberg, and other people who work on multiple comparisons, aim only to correct the type 1 error rate. So they end up with a "correct" P value. It's subject to the same problems as any other P value. In my latest paper, I changed the name from FDR to False Positive Risk (FPR) in an attempt to avoid this misunderstanding.

We've also written a web app to do some of the calculations (after noticing that few people download the R scripts that we provide). It's at https://davidcolquhoun.shinyapps.io/3-calcs-final/ All opinions about itare welcome (please read the Notes tab first).

PS The web calculator now has a new (permanent, I hope) at http://fpr-calc.ucl.ac.uk/ Shiny.io is easy to use, but very expensive if anyone actually uses the app :-(

I've returned to this discussion, now that my second paper on the topic is about to appear in Royal Society Open Science. It is at https://www.biorxiv.org/content/early/2017/08/07/144337

I realise that the biggest mistake that I made in the first paper was to use the term "false discovery rate (FDR)". In the new paper I make it more explicit that I am saying nothing about the multiple comparisons problem. I deal only with the question of how to interpret the P value that's observed in a single unbiased test.

In the latest version, I refer to the probability that the result as the false positive risk (FPR) rather than FDR, in the hope of reducing confusion. I also advocate the reverse Bayesian approach -specify the prior probability that would be needed to ensure an FPR of, say, 5%. If you observe P = 0.05, that comes to 0.87. In other words you'd have to be almost (87%) sure that there was a real effect before doing the experiment to achieve an FPR of 5% (which is what most people still believe, mistakenly, p=0.05 means).

• Dear David, welcome to CrossValidated and thanks for joining in! It seems that we are in agreement about the original @January's question: FDR can only be controlled by an overall BH procedure; if BH is applied in each paper separately, then your arguments still apply. If so, this settles the original question. Regarding my comment about your "too strong" formulations: after reading 147 comments on Mayo's blog, I am hesitant to start another discussion. As I wrote, I mostly agree with your paper, and my objections were only about some formulations. [cont.] Apr 14, 2015 at 14:38
• [...] The first sentence in the abstract is "too strong" exactly for the reasons you listed here: e.g. it assumes point null and it assumes 0.5 prior, but sounds as if it does not assume anything (but I understand that you tried to be provocative). Huge discussion on Mayo's blog shows that many people do not agree that these assumptions are reasonable for actual scientific practice. I have my own objections too, but I do agree with you that these assumptions might accurately describe some scientific fields. And if so, these fields do have a big problem, yes. Apr 14, 2015 at 14:41

A big part of the confusion is that, despite his comments here to the contrary, Colquhoun does NOT define FDR the same way that Benjamini-Hochberg do. It is unfortunate that Colquhoun has attempted to coin a term without first checking to make sure that the term did not already have a well-established, different definition. To make matters worse, Colquhoun defined FDR in precisely the way that the conventional FDR has often been misinterpreted.

In his answer here, Colquhoun defines FDR as "the fraction of positive tests that are false." That is similar to what Benjamini-Hochberg define as the FDP (false discovery proportion, not to be confused with the false discovery rate). Benjamini-Hochberg define FDR as the EXPECTED VALUE of the FDP, with a special stipulation that the FDP is considered as 0 when there are no positive tests (a stipulation that happens to make the FDR equal to the FWER when all nulls are true, and avoids undefinable values due to division by zero).

To avoid confusion, I suggest not worrying about the details in the Colquhoun paper, and instead just taking to heart the big-picture point (which countless others have also made) that the alpha level does not directly correspond to the proportion of significant tests that are Type I errors (whether we're talking about the significant tests in a single study or in several studies combined). That proportion depends not only on alpha, but also on power and on the proportion of tested null hypotheses that are true.