Is there a simple way of explaining why does Benjamini and Hochberg's (1995) procedure actually control the false discovery rate (FDR)? This procedure is so elegant and compact and yet the proof of why it works under independence (appearing in the appendix of their 1995 paper) is not very accessible.

  • 4
    $\begingroup$ in my opinion, the proof of the FDR control presented here is more intuitive (note you're looking for the proof of theorem 2): citeseerx.ist.psu.edu/viewdoc/… There, the argument just relied on noticing that we can use the optional stopping theorem. $\endgroup$ – user795305 Apr 29 '17 at 22:23
  • 3
    $\begingroup$ There's a good lecture from Benjamini on YouTube on the multiple comparisons problem, and the history and logical development of the adjustment methods used to address it. $\endgroup$ – Alexis May 2 '17 at 20:50
  • $\begingroup$ Ramdas et al. (2017) is a very nice recent paper that unifies and generalizes many multiple testing methods, and their Proposition 1(c) implies Theorem 1 in Benjamini & Hochberg (1995). The proof just applies Lemma 1(c) to bound the expectation of the FDP, and this Lemma itself is just proven by very basic multivariate calculus in their appendix. $\endgroup$ – daniel.s May 17 '17 at 22:52
  • 2
    $\begingroup$ Here's another intuitive explanation I found on StatQuest's channel on YouTube: youtube.com/watch?v=K8LQSvtjcEo $\endgroup$ – RobertF Apr 24 '18 at 18:41

Here is some R-code to generate a picture. It will show 15 simulated p-values plotted against their order. So they form an ascending point pattern. The points below the red/purple lines represent significant tests at the 0.1 or 0.2 level. The FDR ist the number of black points below the line divided by the total number of points below the line.

x0 <- runif(10)      #p-values of 10 true null hypotheses. They are Unif[0,1] distributed.
x1 <- rbeta(5,2,30)  # 5 false hypotheses, rather small p-values
xx <- c(x1,x0)
a0 <- sort(xx)
for (i in 1:length(x0)){a0[a0==x0[i]] <- NA}
points(c(1,15), c(1/15 * 0.1 ,0.1), type="l", col="red")
points(c(1,15), c(1/15 * 0.2 ,0.2), type="l", col="purple")

I hope this might give some feeling about the shape the distribution of ordered p-values has. That the lines are correct and not e.g. some parable-shaped curve, has to do with the shape of the order distributions. This has to be calculated explicitly. In fact, the line is just a conservative solution.

  • 1
    $\begingroup$ Would you mind adding set.seed(<some number>) and posting the resulting figure for people who don't read R? $\endgroup$ – gung May 12 '17 at 16:26
  • $\begingroup$ None of the points fall below the line when I run this code... $\endgroup$ – winni2k Nov 2 '17 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.