I'm trying to understand adjustment of confidence intervals for multiple comparisons. I'm referring to this paper by Benjamini: http://rsta.royalsocietypublishing.org/content/367/1906/4255

I don't understand his Figure 1 and the point he's trying to make there.

enter image description here

Figure 1 displays a simulated example of 400 000 realizations of (θi,Yi), where θi are i.i.d. receiving the values $\pm exp(3)$ with equal probability and Yi|θi∼N(θi,1). One can consider θi as the association log-odds ratio and Yi as its estimator. The observations shown in black are the R=58 discoveries produced by the level 0.05 BH procedure, applied to the two-sided p-values pi=2×{1−Φ(|Yi|)}; the remaining observations are shown in grey. The solid lines are the marginal 0.95 confidence intervals $Y_i±Z_{1−0.05/2}$; they cover 0.949 of all 400 000 θi realizations, but only six of the 58 BH discoveries; thus V/R=0.90.

Is $m = 400000$ (the total number of tests)? Is $Y_i$ the test statistic of test $i$? For each test, what is the null hypothesis, what is the alternate hypothesis? Since all the points are generated from the same distribution, aren't the 58 discoveries all false? How are the slant lines generated?


1 Answer 1


The plot shows the false coverage rate with a numerical example.

$400,000$ hypotheses are being tested. $H_0:\theta=0$ for all tests, against a two sided hypothesis. The test statistic for each hypothesis is indeed $Y_i$. All null hypotheses are false, since under $\theta \sim exp(3)$ then $P(\theta=0)=0$.

The slant lines are the FCR-adjusted intervals. They are constructed by deflating the level of the original intervals from $1-\alpha/2$ to $1-(\alpha* 58/400,000) /2$, as the FCR correction implies.

[Edit] The plot is indeed non-standard. Here is how it should be parsed: On the $x$ axis is the observed $y_i$. On the $y$ is the underlying parameter value ($\theta_i$). The confidence intervals for each value of $y_i$ are the vertical distance between matching lines. If a dot is between the lines, it means that the interval has covered its generative parameter. The reported false coverage proportion ($V/R$) is the proportion of selected dots (black) that are not between the lines, meaning that the generative parameter was not covered by the interval constructed on that observation.

See [1] for an explanation (my own) of the procedure.

[1] Rosenblatt, J. D., and Y. Benjamini. “Selective Correlations; Not Voodoo.” NeuroImage 103 (December 2014): 401–10. doi:10.1016/j.neuroimage.2014.08.023.

  • $\begingroup$ Thanks JohnRos! I get the first half of your answer. But I'm still finding it hard to understand how the original confidence intervals were constructed in the first place i.e. how were the solid slant lines drawn? Before posting this question, I searched the forum and found your paper and read it - it helped me a lot. I get the FCR adjustment part, but I'm confused about the slant lines. $\endgroup$
    – elexhobby
    Sep 28, 2015 at 4:56
  • $\begingroup$ I think I understand the slant lines - for each Y (on the X axis) the lower line is plotted at (Y-1.96) and the upper line at (Y+1.96). Is that the right way of thinking about them or should I be looking at them for each value of $\theta$? Either way, I still don't get why the 6 discoveries included between the solid lines are false discoveries. $\endgroup$
    – elexhobby
    Sep 28, 2015 at 13:45
  • $\begingroup$ @elexhobby: edited my answer with more details. $\endgroup$
    – JohnRos
    Sep 28, 2015 at 14:00
  • 1
    $\begingroup$ Ah! Thanks! That makes it clear. I guess the point the authors are trying to make is: if you look at all points, the confidence intervals cover their respective parameter with 0.949 probability, but if you look only at the selected points, this drops down to 0.1. $\endgroup$
    – elexhobby
    Sep 28, 2015 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.