5
$\begingroup$

I am wondering if the local FDR in Efron's literature is different than the FDR associated with Benjamini-Hochberg and if it is perhaps talking about something else.

$\endgroup$
2
  • $\begingroup$ Do you hand a link to an example of the specific use you are querying? $\endgroup$
    – ReneBt
    Mar 9, 2020 at 3:22
  • $\begingroup$ BH stepdown doesn't control the FDR, it controls the FWER. $\endgroup$
    – AdamO
    Feb 26, 2021 at 15:52

2 Answers 2

2
$\begingroup$

Yes, they are different. Local FDR in Efron's literature is easier to understand, and it's a Bayesian idea. We suppose the parameter $\theta$ has a prior distribution, say \begin{equation*} \theta \sim \begin{cases} 0 & \text{with probability } \pi_0 \\ f_1(\theta) & \text{with probability } 1-\pi_0 \end{cases} \end{equation*} We also suppose that we observe a statistic $z \sim g(\theta)$. The local FDR is basically the posterior probability $p(\theta = 0 | z)$. There's also a related concept in Efron's literature, the Bayes FDR, which is basically $p(\theta=0|z \in \mathcal{Z})$ where $\mathcal{Z}$ is a rejection region (e.g. when the p-value given $z$ is less than 0.05).

Bejamini-Hochberg is more difficult to understand, cos it's a frequentist concept. Suppose you run a large number ($n$) of tests, to test whether $\theta_1=0, \theta_2=0, \ldots, \theta_n=0$. Then, imagine you repeat this set of tests many (infinitely many) times. With each repeat, although $\boldsymbol{\theta}=(\theta_1, \theta_2, \ldots, \theta_n)$ stays the same, $\boldsymbol{z}=(z_1, z_2, \ldots, z_n)$ are generated anew from $z_i \sim g(\theta_i)$. The "false discovery rate" for each repeat $k$ corresponds to $S(k) = Pr(z_{ik} \in \mathcal{Z}, \theta_i = 0 | z_{ik} \in \mathcal{Z}) \equiv \dfrac{\#(z_{ik} \in \mathcal{Z}, \theta_i = 0)}{\#(z_{ik} \in \mathcal{Z})}$, with $Pr(z_{ik} \in \mathcal{Z}, \theta_i = 0 | z_{ik} \in \mathcal{Z}) = 0$ if $\#(z_{ik} \in \mathcal{Z}) = 0$, and $\#$ denotes "counts of". Note that $S(k)$ is a random variable that varies with $k$. In Benjamini Hochberg's definition, $FDR=\mathbb{E}(S(k))$. They are saying if you follow their procedure, given the assumptions (e.g. independence of $z_i$), $FDR \leq \alpha$.

$\endgroup$
3
  • $\begingroup$ No wonder 'Bejamini-Hochberg is more difficult to understand' - your explanation is very confusing BH is simply that you rank each test and adjust the p-value by number of tests and check the first test is below the adjusted p-value. If it is you drop that test, adjust the threshold for the lower number of tests and repeat with one less test until you get to the highest p value that fails against the adjusted threshold. $\endgroup$
    – ReneBt
    Mar 9, 2020 at 10:57
  • 1
    $\begingroup$ You are talking about BH as a procedure. The question concerns the definition of FDR in the BH paper, which they use to justify their procedure. $\endgroup$
    – Tim Mak
    Mar 10, 2020 at 2:41
  • $\begingroup$ My point is you can present it simply or complicated, the same as any mathematical operation. If you present the complicated version then it is biased towards being difficult to understand, so difficulty becomes a self fulfilling prophecy. Your answer reads very well without the labels regarding difficulty. $\endgroup$
    – ReneBt
    Mar 10, 2020 at 9:24
1
$\begingroup$

Efron has a very good explanation of this in local FDR vs BH FDR. Figure 3 in that reference gives a very good geometric view of how the two FDR methods differ. Bonferroni is returned to below.

Fundamental concept of each test

Local fdr (I'm using the capitalisation in Efron's paper) is based on determining the likelihood of the p value being a false positive under null vs the sum of it being a true positive under the prior or a false positive under null. It does this for each test hence the use of 'local' in the name to differentiate it from BH.

BH is based on making long run assumptions about the likelihood of the null giving random false positives. BH recognises that independent probabilities combine using OR logic, so multiple test probabilities sum together. Thus the probability of a false positive occurring within a suite of independent tests is the sum of the probabilities in each each test. In practical use rather than adjust the p-value or z-score itself one common implementation of BH is that the threshold for acceptance or rejection is adjusted to reflect the sum of probable false positives across the suite of tests. The mathematical explanation of this implementation of BH made most sense to me and is firmly grounded in set theory mathematics, in the form of positive regression dependence on a subset of one. here's my understanding of it. To address the point raised in comments about the relevance of this procedure to the question, the implementation may not be what Benjamini and Hochberg originally envisaged, but it is a mathematically sound way of achieving their intent and I found it comprehensible so others may too.

Consequences of differences

This implies that, to quote Efron:

The local nature of fdr(z) is an advantage in interpreting results for individual cases.

Efron states (eq 2.11 in reference) that the relationship at low values in BH is $$fdr(z) ˙= Fdr(z)/α$$ where $fdr$ is the local FDR and $Fdr$ is the BH. The full relationship is given by Eq 2.8 $$Fdr(z) = E_f [fdr(Z)|Z ≤ z],$$ where z is the actual z-score realised and Z is the rejection region. I.e. the BH Fdr is the expectation of the fdr given z exceeds the threshold. BH does not worry about how much an individual test exceeds the adjusted threshold, just whether it does or not.

BH requires that each statistic be independent (subject to OR logic), otherwise false positives will be correlated and the adjusted threshold for each test is no longer accurate. Strongly correlated tests need to be handled differently depending on the nature of the correlation. In contrast the local FDR is defined based on the prior specific to each test and so the requirement of independence can be relaxed (I don't see this stated so I am guessing this is because any correlation will be baked into the priors and so explicitly accounted for), but for useful distributions to be defined then the number of tests must be large. To quote Efron:

$N$ must be large for local fdr calculations, at least in the hundreds, but the $z_i$ need not be independent.

This is simply because to build robust data based metrics you need enough data to provide a robust estimate of derived parameters.

Bonferroni

Bonferroni differs greatly in its intent from Efron or BH, it controls for the possibility of at least one false positive and is a family-wise error rate controllers (all tests treated uniformly as a common family). Bonferroni more tightly controls the risk of any false positives at the expense of power. FDR methods attempt to provide a more nuanced handling of false positive risk (evaluating risk evaluation for each test individually), providing better power at the expense of a higher risk of false positives.

$\endgroup$
2
  • $\begingroup$ The so-called BH FDR you referred to is not actually the FDR referred to by Benjamini and Hochberg (1995). As I pointed out in my answer, the FDR in BH1995 is an unobserved r.v.. The main goal of BH1995 is to control for this FDR. Your "BH FDR" is a method for estimating the sample's false discovery rate (which, confusingly, is not BH1995's FDR either), inspired, no doubt, by BH1995's procedure for FDR control. As Efron said in his paper, this procedure for estimation of FDR was first described by Storey (2002) and Efron and Tibshirani (2002). $\endgroup$
    – Tim Mak
    Mar 13, 2020 at 10:36
  • $\begingroup$ Thanks for your feedback, it helps me understand you reaction to my comments under your answer. I don't see a conflict between my answer and your comment. I am discussing BH-FDR on the basis of set theory implementation (PDRS1), which while procedural is still mathematical (modern maths is founded on set theory). I choose this perspective as it provides additional perspective wrt to your answer and it is the one that helped me understand BH-FDR. I appreciate the procedure doesn't give a precise adjusted probability, but it achieves the same goal for a predefined $\alpha$. $\endgroup$
    – ReneBt
    Mar 13, 2020 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.