6
$\begingroup$

Background: the Benjamini-Hochberg procedure is a method for correcting for multiple p-values. (If you set p<.05, and you do 20 tests, then one test is likely to be a false positive. The B-H procedure corrects for this.)

When using this procedure, not only do you set a p value (usually .05), you also set a q value - the False Discovery Rate (or FDR). This does not necessarily need to be .05.

This CrossValidated thread discusses the correctness of choosing the q value after viewing your data, but it does not answer my question.

My question: what is usually considered a reasonable q value? I've seen everything from .05 to .20 used in literature, but I have found no guidance on what are the highest values that may be considered "conservative" and what is considered a stretch.

$\endgroup$

3 Answers 3

3
$\begingroup$

FDRs are commonly much greater than p-values, and recall the lowest FDR value represents a list of features and not one feature, like a p-value does. So if the lowest value of FDR is for example 0.15 for a list of 15 features(genes), then at the very least you have to publish the list and state that the FDR is 0.15. If reviewers expect to see lower FDR's then there's not much of an alternative. Only lower p-values can drive FDR lower. If you can generate a list of 10-15 or 20-30 features whose FDR is 0.05, then you should have no problem publishing this in the peer-reviewed literature. However, FDR values of 0.1, 0.15, and 0.2 and greater are frowned upon -- i.e. are not considered "good." This does not mean however, that you can't publish a list of features when FDR is not 0.05 or is greater than 0.05 -- it's a matter of the personal-experience-based FDR threshold assumed by the reviewer, laboratory, or journal.

$\endgroup$
3
  • 2
    $\begingroup$ Thank you, that is helpful. is it common to publish with FDR somewhere between .05 and .1? $\endgroup$ Dec 22, 2016 at 21:57
  • $\begingroup$ Yes, that should work! $\endgroup$
    – user32398
    Dec 23, 2016 at 0:27
  • $\begingroup$ I think we should avoid confusion in terminology. There is a cutoff value chosen a priori (we can define it as Q or q*), so that the FDR must not overcome it. Then, there is a single q-value for each comparison, and we need at least one of them to be below such cutoff. So, we have a list of q-values, not of FDR values. $\endgroup$ Jul 5, 2023 at 20:30
2
$\begingroup$

In my experience, there is no such thing on what the "best" FDR is. The "best" FDR depends on your desired statistical power and type I error. In genetics, FDR 0.05 is common, but this is not mandatory.

When you report your results, you should make FDR to 0.1 or 0.05 unless you have a good reason not to. Your report will be consistent to the literature because other people also use 0.1 or 0.05. If you set your FDR to something like 0.0456, your paper might be rejected for inconsistency.

$\endgroup$
0
$\begingroup$

I don’t think you should set both a p-value and Q (the upper value of FDR). The p-values are calculated and, with the BH correction, statistical significance depends on both the set of p-values and such Q. The link you report actually sets a Q of 0.25 (and defends such choice even in case of all p-values between 0.10 and 0.24): I’ve never seen a Q above that. The reason for discrepancies across fields, in my opinion, may be understood by reading the original paper: https://rss.onlinelibrary.wiley.com/doi/10.1111/j.2517-6161.1995.tb02031.x The FDR is given by the number of false positives on the number of significant findings. However, a crucial point is that, in case of no significant findings, the FDR is set to 0. This makes the FDR equal to the p-value in the single-test case (when the null hypothesis is true). Of course, this is not interesting because it’s the degenerate case. However, the point is that, when the number of tests is low, or there’s the possibility that all null hypotheses are true, using an alpha above 0.05 would be suspect. For example, with only two tests, you would consider two p-values between 0.05 and 0.1 as evidence of a double effect, and you would consider a p-value of 0.05 in one test as significant regardless of the value of the other test: this would be even less conservative than ignoring the fact you’re doing two tests. If, instead, it is not realistic to assume that all null hypotheses are true, it may make sense to use a higher Q, since the possibility of no significant findings becomes less realistic. Moreover, the formula in the original paper shows that you just select an upper bound for FDR, but that actual FDR is below the product between the chosen Q and the proportion of null hypotheses that are true ( $E(Q) \leq \frac{m_o}{m}q* \leq q*$, in the paper’s terminology). This implies that, the higher the ratio of false null hypotheses, the lower the real FDR. This explains why, in situations where a plethora of tests are performed, like in genetics, one may choose higher Qs. Consider however that use of a Q of 0.05 allows you to express q-values as BH-adjusted p-values, and that you’ll be sure to have both 1) BH-adjusted p-values more conservative than unadjusted p-values and 2) BH-adjusted CIs for significant findings (see, for example, here: https://onlinelibrary.wiley.com/doi/10.1111/psyp.12616) with a higher coverage than 95% and not including the null. Instead, with a Q above 0.05, you could find significance in comparisons with a p-value above 0.05 (and with a 95% unadjusted CI including the null). Moreover, referees might not accept a Q above 0.05, for analogy between FDR and type-I error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.