Benjamini & Hochberg define false discovery rate in the same way that I do, as the fraction of positive tests that are false positives. So if you use their procedure for multiple comparisons you control FDR properly. It's worth noting, though, that there are quite a lot of variants on the B-H method. Benjamini's seminars at Berkeley are on Youtube, and well worth watching:
I'm not sure why @amoeba says "This is formulated too strongly and can actually be misleading". I'd be interested to know why he/she thinks that. The most persuasive argument comes from the simulated t tests (section 6). That mimics what almost everyone does in practice and it shows that if you observe P close to 0.047, and claim to have made a discovery, you'll be wrong at least 26% of the time. What can go wrong?
Of course, I should not describe this as a minimum. It's what you get if you assume that there's a 50% chance of the there being a real effect. Of course if you assume that most of your hypotheses are correct in advance, then you can get a lower FDR than 26%, but can you imagine the hilarity that would greet a claim that you'd made a discovery on the basis of the assumption that you were 90% sure in advance that your conclusion would be true. 26% is the minimum FDR given that it isn't a reasonable basis for inference to assume any prior probability greater than 0.5.
Given that hunches frequently don't stand up when tested, it could well be that there is only a 10% chance of any particular hypothesis being true, and in that case the FDR would be a disastrous 76%.
It's true that all this is contingent on the null hypothesis being that there is zero difference (the so called point null). Other choices can give different results. But the point null is what almost everyone uses in real life (though the may not be aware of it). Furthermore the point null seems to me to be entirely appropriate thing to use. It's sometimes objected that true differences are never exactly zero. I disagree. We want to tell whether are not our results are distinguishable from the case where both groups are given identical treatments, so the true difference is exactly zero. If we decide that out data are not compatible with that view, we go on to estimate the effect size. and at that point we make the separate judgment about whether the effect, though real, is big enough to be important in practice. There is some vigorous discussion of these topics on Deborah Mayo's blog.
@amoeba Thanks for you response.
What the discussion on Mayo's blog shows is mostly that Mayo doesn't agree with me, though she hasn't made clear why, to me at least). Stephen Senn points out correctly that you can get a different answer if you postulate a different prior distribution. That seems to me to be interesting only to subjective Bayesians.
It's certainly irrelevant to everyday practice which always assumes a point null. And as I explained, that seems to me to be a perfectly sensible thing to do.
Many professional statisticians have come to conclusions much the same as mine. Try Sellke & Berger, and Valen Johnson (refs in my paper). There is nothing very controversial (or very original) about my claims.
Your other point, about assuming a 0.5 prior, doesn't seem to me to be an assumption at all. As I explained above, anything above 0.5 woold be unacceptable in practice. And anything below 0.5 makes the false discovery rate even higher (eg 76% if prior is 0.1). Therefore it's perfectly reasonable to say that 26% is the minimum false discovery rate that you can expect if you observe P = 0.047 in a single experiment.
I've been thinking more about this question. My definition of FDR is the same as Benjamini's -the fraction of positive tests that are false. But it is applied to a quite different problem, the interpretation of a single test. With hindsight it might have been better if I'd picked a different term.
In the case of a single test, B&H leaves the P value unchanged, so it does not say anything about the false discovery rate in the sense that I use the term.
es of course you are right. Benjamini & Hochberg, and other people who work on multiple comparisons, aim only to correct the type 1 error rate. So they end up with a "correct" P value. It's subject to the same problems as any other P value.
In my latest paper, I changed the name from FDR to False Positive Risk (FPR) in an attempt to avoid this misunderstanding.
We've also written a web app to do some of the calculations (after noticing that few people download the R scripts that we provide). It's at https://davidcolquhoun.shinyapps.io/3-calcs-final/ All opinions about itare welcome (please read the Notes tab first).
PS The web calculator now has a new (permanent, I hope) at http://fpr-calc.ucl.ac.uk/
Shiny.io is easy to use, but very expensive if anyone actually uses the app :-(
I've returned to this discussion, now that my second paper on the topic is about to appear in Royal Society Open Science. It is at https://www.biorxiv.org/content/early/2017/08/07/144337
I realise that the biggest mistake that I made in the first paper was to use the term "false discovery rate (FDR)". In the new paper I make it more explicit that I am saying nothing about the multiple comparisons problem. I deal only with the question of how to interpret the P value that's observed in a single unbiased test.
In the latest version, I refer to the probability that the result as the false positive risk (FPR) rather than FDR, in the hope of reducing confusion. I also advocate the reverse Bayesian approach -specify the prior probability that would be needed to ensure an FPR of, say, 5%. If you observe P = 0.05, that comes to 0.87. In other words you'd have to be almost (87%) sure that there was a real effect before doing the experiment to achieve an FPR of 5% (which is what most people still believe, mistakenly, p=0.05 means).
