Benjamini-Hochberg: choosing the False Discovery Rate / q value

Question

Background: the Benjamini-Hochberg procedure is a method for correcting for multiple p-values. (If you set p<.05, and you do 20 tests, then one test is likely to be a false positive. The B-H procedure corrects for this.)

When using this procedure, not only do you set a p value (usually .05), you also set a q value - the False Discovery Rate (or FDR). This does not necessarily need to be .05.

This CrossValidated thread discusses the correctness of choosing the q value after viewing your data, but it does not answer my question.

My question: what is usually considered a reasonable q value? I've seen everything from .05 to .20 used in literature, but I have found no guidance on what are the highest values that may be considered "conservative" and what is considered a stretch.

Answer 1

FDRs are commonly much greater than p-values, and recall the lowest FDR value represents a list of features and not one feature, like a p-value does. So if the lowest value of FDR is for example 0.15 for a list of 15 features(genes), then at the very least you have to publish the list and state that the FDR is 0.15. If reviewers expect to see lower FDR's then there's not much of an alternative. Only lower p-values can drive FDR lower. If you can generate a list of 10-15 or 20-30 features whose FDR is 0.05, then you should have no problem publishing this in the peer-reviewed literature. However, FDR values of 0.1, 0.15, and 0.2 and greater are frowned upon -- i.e. are not considered "good." This does not mean however, that you can't publish a list of features when FDR is not 0.05 or is greater than 0.05 -- it's a matter of the personal-experience-based FDR threshold assumed by the reviewer, laboratory, or journal.

Answer 2

In my experience, there is no such thing on what the "best" FDR is. The "best" FDR depends on your desired statistical power and type I error. In genetics, FDR 0.05 is common, but this is not mandatory.

When you report your results, you should make FDR to 0.1 or 0.05 unless you have a good reason not to. Your report will be consistent to the literature because other people also use 0.1 or 0.05. If you set your FDR to something like 0.0456, your paper might be rejected for inconsistency.

Answer 3

I don’t think you should set both a p-value and Q (the upper value of FDR). The p-values are calculated and, with the BH correction, statistical significance depends on both the set of p-values and such Q. The link you report actually sets a Q of 0.25 (and defends such choice even in case of all p-values between 0.10 and 0.24): I’ve never seen a Q above that. The reason for discrepancies across fields, in my opinion, may be understood by reading the original paper: https://rss.onlinelibrary.wiley.com/doi/10.1111/j.2517-6161.1995.tb02031.x The FDR is given by the number of false positives on the number of significant findings. However, a crucial point is that, in case of no significant findings, the FDR is set to 0. This makes the FDR equal to the p-value in the single-test case (when the null hypothesis is true). Of course, this is not interesting because it’s the degenerate case. However, the point is that, when the number of tests is low, or there’s the possibility that all null hypotheses are true, using an alpha above 0.05 would be suspect. For example, with only two tests, you would consider two p-values between 0.05 and 0.1 as evidence of a double effect, and you would consider a p-value of 0.05 in one test as significant regardless of the value of the other test: this would be even less conservative than ignoring the fact you’re doing two tests. If, instead, it is not realistic to assume that all null hypotheses are true, it may make sense to use a higher Q, since the possibility of no significant findings becomes less realistic. Moreover, the formula in the original paper shows that you just select an upper bound for FDR, but that actual FDR is below the product between the chosen Q and the proportion of null hypotheses that are true ( $E(Q) \leq \frac{m_o}{m}q* \leq q*$, in the paper’s terminology). This implies that, the higher the ratio of false null hypotheses, the lower the real FDR. This explains why, in situations where a plethora of tests are performed, like in genetics, one may choose higher Qs. Consider however that use of a Q of 0.05 allows you to express q-values as BH-adjusted p-values, and that you’ll be sure to have both 1) BH-adjusted p-values more conservative than unadjusted p-values and 2) BH-adjusted CIs for significant findings (see, for example, here: https://onlinelibrary.wiley.com/doi/10.1111/psyp.12616) with a higher coverage than 95% and not including the null. Instead, with a Q above 0.05, you could find significance in comparisons with a p-value above 0.05 (and with a 95% unadjusted CI including the null). Moreover, referees might not accept a Q above 0.05, for analogy between FDR and type-I error.

****************** EDIT 25/7/2024 *************************************

Yesterday I thought of a method to decide the FDR ex-post, i.e.: "select a FDR such that the k significant p-values are such that the remaining (n-k) p-values are uniformly distributed, while, if we added the least significant of them, those (n-k+1) p-values would not be significant anymore". The problem I found however is that (apart from the possible lack of power) such reasoning only holds in case of independence of tests.