Is there a simple way of explaining why does Benjamini and Hochberg's (1995) procedure actually control the false discovery rate (FDR)? This procedure is so elegant and compact and yet the proof of why it works under independence (appearing in the appendix of their 1995 paper) is not very accessible.
$\begingroup$
$\endgroup$
4
Michael R. Chernick
43.2k28 gold badges85 silver badges159 bronze badges
asked Apr 29, 2017 at 20:48
-
5$\begingroup$ in my opinion, the proof of the FDR control presented here is more intuitive (note you're looking for the proof of theorem 2): citeseerx.ist.psu.edu/viewdoc/… There, the argument just relied on noticing that we can use the optional stopping theorem. $\endgroup$– user795305Commented Apr 29, 2017 at 22:23
-
4$\begingroup$ There's a good lecture from Benjamini on YouTube on the multiple comparisons problem, and the history and logical development of the adjustment methods used to address it. $\endgroup$– AlexisCommented May 2, 2017 at 20:50
-
$\begingroup$ Ramdas et al. (2017) is a very nice recent paper that unifies and generalizes many multiple testing methods, and their Proposition 1(c) implies Theorem 1 in Benjamini & Hochberg (1995). The proof just applies Lemma 1(c) to bound the expectation of the FDP, and this Lemma itself is just proven by very basic multivariate calculus in their appendix. $\endgroup$– daniel.sCommented May 17, 2017 at 22:52
-
2$\begingroup$ Here's another intuitive explanation I found on StatQuest's channel on YouTube: youtube.com/watch?v=K8LQSvtjcEo $\endgroup$– RobertFCommented Apr 24, 2018 at 18:41
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Here is some R-code to generate a picture. It will show 15 simulated p-values plotted against their order. So they form an ascending point pattern. The points below the red/purple lines represent significant tests at the 0.1 or 0.2 level. The FDR ist the number of black points below the line divided by the total number of points below the line.
x0 <- runif(10) #p-values of 10 true null hypotheses. They are Unif[0,1] distributed.
x1 <- rbeta(5,2,30) # 5 false hypotheses, rather small p-values
xx <- c(x1,x0)
plot(sort(xx))
a0 <- sort(xx)
for (i in 1:length(x0)){a0[a0==x0[i]] <- NA}
points(a0,col="red")
points(c(1,15), c(1/15 * 0.1 ,0.1), type="l", col="red")
points(c(1,15), c(1/15 * 0.2 ,0.2), type="l", col="purple")
I hope this might give some feeling about the shape the distribution of ordered p-values has. That the lines are correct and not e.g. some parable-shaped curve, has to do with the shape of the order distributions. This has to be calculated explicitly. In fact, the line is just a conservative solution.
-
1$\begingroup$ Would you mind adding
set.seed(<some number>)and posting the resulting figure for people who don't read R? $\endgroup$ Commented May 12, 2017 at 16:26 -
$\begingroup$ None of the points fall below the line when I run this code... $\endgroup$– winni2kCommented Nov 2, 2017 at 14:51