Benjamini-Hochberg: choosing the False Discovery Rate / q value

Question

Background: the Benjamini-Hochberg procedure is a method for correcting for multiple p-values. (If you set p<.05, and you do 20 tests, then one test is likely to be a false positive. The B-H procedure corrects for this.)

When using this procedure, not only do you set a p value (usually .05), you also set a q value - the False Discovery Rate (or FDR). This does not necessarily need to be .05.

This CrossValidated thread discusses the correctness of choosing the q value after viewing your data, but it does not answer my question.

My question: what is usually considered a reasonable q value? I've seen everything from .05 to .20 used in literature, but I have found no guidance on what are the highest values that may be considered "conservative" and what is considered a stretch.

Answer 1

FDRs are commonly much greater than p-values, and recall the lowest FDR value represents a list of features and not one feature, like a p-value does. So if the lowest value of FDR is for example 0.15 for a list of 15 features(genes), then at the very least you have to publish the list and state that the FDR is 0.15. If reviewers expect to see lower FDR's then there's not much of an alternative. Only lower p-values can drive FDR lower. If you can generate a list of 10-15 or 20-30 features whose FDR is 0.05, then you should have no problem publishing this in the peer-reviewed literature. However, FDR values of 0.1, 0.15, and 0.2 and greater are frowned upon -- i.e. are not considered "good." This does not mean however, that you can't publish a list of features when FDR is not 0.05 or is greater than 0.05 -- it's a matter of the personal-experience-based FDR threshold assumed by the reviewer, laboratory, or journal.

Answer 2

In my experience, there is no such thing on what the "best" FDR is. The "best" FDR depends on your desired statistical power and type I error. In genetics, FDR 0.05 is common, but this is not mandatory.

When you report your results, you should make FDR to 0.1 or 0.05 unless you have a good reason not to. Your report will be consistent to the literature because other people also use 0.1 or 0.05. If you set your FDR to something like 0.0456, your paper might be rejected for inconsistency.