Understanding False Discovery Rate

Question

I'm trying to understand adjustment of confidence intervals for multiple comparisons. I'm referring to this paper by Benjamini: http://rsta.royalsocietypublishing.org/content/367/1906/4255

I don't understand his Figure 1 and the point he's trying to make there.

Figure 1 displays a simulated example of 400 000 realizations of (θi,Yi), where θi are i.i.d. receiving the values $\pm exp(3)$ with equal probability and Yi|θi∼N(θi,1). One can consider θi as the association log-odds ratio and Yi as its estimator. The observations shown in black are the R=58 discoveries produced by the level 0.05 BH procedure, applied to the two-sided p-values pi=2×{1−Φ(|Yi|)}; the remaining observations are shown in grey. The solid lines are the marginal 0.95 confidence intervals $Y_i±Z_{1−0.05/2}$; they cover 0.949 of all 400 000 θi realizations, but only six of the 58 BH discoveries; thus V/R=0.90.

Is $m = 400000$ (the total number of tests)? Is $Y_i$ the test statistic of test $i$? For each test, what is the null hypothesis, what is the alternate hypothesis? Since all the points are generated from the same distribution, aren't the 58 discoveries all false? How are the slant lines generated?

Answer 1

The plot shows the false coverage rate with a numerical example.

$400,000$ hypotheses are being tested. $H_0:\theta=0$ for all tests, against a two sided hypothesis. The test statistic for each hypothesis is indeed $Y_i$. All null hypotheses are false, since under $\theta \sim exp(3)$ then $P(\theta=0)=0$.

The slant lines are the FCR-adjusted intervals. They are constructed by deflating the level of the original intervals from $1-\alpha/2$ to $1-(\alpha* 58/400,000) /2$, as the FCR correction implies.

[Edit] The plot is indeed non-standard. Here is how it should be parsed: On the $x$ axis is the observed $y_i$. On the $y$ is the underlying parameter value ($\theta_i$). The confidence intervals for each value of $y_i$ are the vertical distance between matching lines. If a dot is between the lines, it means that the interval has covered its generative parameter. The reported false coverage proportion ($V/R$) is the proportion of selected dots (black) that are not between the lines, meaning that the generative parameter was not covered by the interval constructed on that observation.

See [1] for an explanation (my own) of the procedure.

[1] Rosenblatt, J. D., and Y. Benjamini. “Selective Correlations; Not Voodoo.” NeuroImage 103 (December 2014): 401–10. doi:10.1016/j.neuroimage.2014.08.023.