Can someone explain how FDR procedures are able to estimate an FDR without a model / assumption of the base rate of true positives?


I think that's a really good question; too many people use the Benjamini-Hochberg procedure (abbreviated BH; possibly the most popular procedure to control the FDR) as a black box. Indeed there is an underlying assumption it makes on the statistics and it is nicely hidden in the definition of the p-values!

For a well-defined p-value $P$ it holds that $P$ is uniformly distributed ($P\sim U[0,1]$) under the null hypothesis. Sometimes it might even be that $\Pr[P\leq t] \leq t$, i.e. that $P$ is stochastically smaller than uniform, but this only makes the procedures more conservative (and therefore still valid). Thus, by calculating your p-values, using a t-test or really any test of your choice, you are providing the information about the distribution under the null hypothesis.

But notice here that I kept talking about the null hypothesis; so what you mentioned about knowledge of the base rate of true positives is not needed, you only need knowledge of the base rate of false positives! Why is this?

Let $R$ denote the number of all the rejected (positive) hypotheses and $V$ the false positives, then:

$$ \text{FDR} = \mathbb E\left[\frac{V}{\max(R,1)}\right] \approx \frac{\mathbb E[V]}{\mathbb E[R]}$$

So to estimate the FDR you need a way of estimating $\mathbb E[R]$, $\mathbb E[V]$. We will now look at decision rules which reject all p-values $\leq t$. To make this clear in the notation I will also write $FDR(t),R(t),V(t)$ for the corresponding quantities/random variables of such a procedure.

Since $\mathbb E[R(t)]$ is just the expectation of the total number of rejections, you can unbiasedly estimate it by the number of rejections you observe, so $\mathbb E[R(t)] \approx R(t)$, i.e. simply by counting how many of your p-values are $\leq t$.

Now what about $\mathbb E[V]$? Well assume $m_0$ of your $m$ total hypotheses are null hypotheses, then by the uniformity (or sub-uniformity) of the p-values under the null you get:

$$\mathbb E[V(t)] = \sum_{i \text{ null}} \Pr[P_i \leq t] \leq m_0 t$$

But we still do not know $m_0$, but we know that $m_0 \leq m$, so a conservative upper bound would just be $\mathbb E[V(t)] \leq m t$. Therefore, since we just need an upper bound on the number of false positives, it is enough that we know their distribution! And this is exactly what the BH procedure does.

So, while Aarong Zeng's comment that "the BH procedure is a way to control the FDR at the given level q. It's not about estimating the FDR" is not false, it is also highly misleading! The BH procedure actually does estimate the FDR for each given threshold $t$. And then it chooses the biggest threshold, such that the estimated FDR is below $\alpha$. Indeed the "adjusted p-value" of hypothesis $i$ is essentially just an estimate of the FDR at the threshold $t=p_i$ (up to isotonization). I think the standard BH algorithm hides this fact a bit, but it is easy to show the equivalence of these two approaches (also called the "equivalence theorem" in the Multiple testing literature).

As a final remark, there do exist methods such as Storey's procedure which even estimate $m_0$ from the data; this can increase power by a tiny bit. Also in principle you are right, one could also model the distribution under the alternative (your true positive base rate) to get more powerful procedures; but so far the multiple testing research has mainly focused on maintaining control of type-I error rather than maximizing power. One difficulty would also be that in many cases each of your true alternatives will have a different alternative distribution (e.g. different power for different hypotheses), while under the null all p-values have the same distribution. This makes the modelling of the true positive rate even more difficult.

| cite | improve this answer | |
  • 3
    $\begingroup$ +1 Presumably "BH" refers to Benjamini-Hochberg. (It's always a good idea to spell out acronyms, lest people misunderstand.) Welcome to our site! $\endgroup$ – whuber Oct 23 '15 at 14:23
  • 1
    $\begingroup$ Thanks! Also yes you are right, I edited my post to reflect that. $\endgroup$ – air Oct 23 '15 at 14:32

As suggested by @air, the Benjamini-Hochberg (BH) procedure guarantees FDR control. It does not aim at estimating it. It thus requires a mere weak dependence assumption between test statistics. [1,2]

Methods that aim at estimating the FDR [e.g. 3,4,5] do require some assumptions on the generative process in order to estimate it. They typically assume test statistics are independent. They will also assume something on the null distribution of the test statistics. Departures from this null distribution, together with the independence assumption, can thus be attributed to effects, and the FDR may be estimated.

Note that these ideas reappear in the semi-supervised novelty detection literature. [6].

[1] Benjamini, Y., and Y. Hochberg. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” JOURNAL-ROYAL STATISTICAL SOCIETY SERIES B 57 (1995): 289–289.

[2] Benjamini, Y., and D. Yekutieli. “The Control of the False Discovery Rate in Multiple Testing under Dependency.” ANNALS OF STATISTICS 29, no. 4 (2001): 1165–88.

[3] Storey, J.D. “A Direct Approach to False Discovery Rates.” Journal Of The Royal Statistical Society Series B 64, no. 3 (2002): 479–98. doi:10.1111/1467-9868.00346.

[4] Efron, B. “Microarrays, Empirical Bayes and the Two-Groups Model.” Statistical Science 23, no. 1 (2008): 1–22.

[5] Jin, Jiashun, and T. Tony Cai. “Estimating the Null and the Proportion of Nonnull Effects in Large-Scale Multiple Comparisons.” Journal of the American Statistical Association 102, no. 478 (June 1, 2007): 495–506. doi:10.1198/016214507000000167.

[6] Claesen, Marc, Jesse Davis, Frank De Smet, and Bart De Moor. “Assessing Binary Classifiers Using Only Positive and Unlabeled Data.” arXiv:1504.06837 [cs, Stat], April 26, 2015. http://arxiv.org/abs/1504.06837.

| cite | improve this answer | |
  • 1
    $\begingroup$ +1 though my main point from that paragraph was that the BH procedure actually does suggest a way of estimating the FDR (albeit a bit conservatively) and in fact does estimate it to arrive at the final rejection threshold. Its algorithmic definition as a step-up procedure in reference [1] obscures this, but at the end of the day estimation of FDR is exactly what the BH procedure does!! (Efron often makes that point, but also see Section 4. "A connection between the two approaches" in your reference [3].) $\endgroup$ – air Oct 23 '15 at 15:46
  • 2
    $\begingroup$ You are right that following [3, Eq.2.5], one may see the BH procedure as using a conservative estimate of the FDR with $p_0=1$. $\endgroup$ – JohnRos Oct 23 '15 at 15:53

When the true underlying model is unknown, we cannot compute the FDR, but can estimate the FDR value by permutation test. Basically the permutation test procedure is just doing the hypothesis test multiple times by change the outcome variable vector with its permutations. It can also be done based on the permutations of the samples, but not as common as the former one.

The paper here reviews the standard permutation procedure for FDR estimation, and also proposed a new FDR estimator. It should be able to address your question.

| cite | improve this answer | |
  • 3
    $\begingroup$ The most common procedure like BH doesn't use a permutation test. What does it use? Also, permutation tests usually provide a distribution under the null, doesn't an FDR estimate require models of both the null and alternative as well as the underlying relative proportion of each? $\endgroup$ – user4733 Nov 17 '14 at 20:16
  • $\begingroup$ First, the BH procedure is a way to control the FDR at the given level $q$. It's not about estimating the FDR. Second, the permutation tests are conducted under the null of all hypotheses. I am not sure what you mean by "require models of both the null and alternative as well as the underlying relative proportion of each". But when you set up your hypotheses, you already have your null and alternative pairs. Does this make sense? $\endgroup$ – Aaron Zeng Nov 19 '14 at 1:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.