Correction for multiple testing on a modest number of tests (10-20) with FDR?

Question

False Discovery Rate (Benjamini-Hochberg) is typically used on 'Big Data', such as genetic studies using 100s of tests. But can it also be used on smaller numbers of tests? For example, looking at the outcomes of two groups (males vs females) on, say, 10-20 different questionnaires. Does the FDR procedure lose value/meaning/power in these cases?

Answer 1

I see people confusing this all the time, also in this forum. I think this is caused to a large extent because in practice Benjamini-Hochberg's procedure is used as a synonym of False Discovery Rate (and as a black-box for "adjusting" p-values as requested by reviewers for their papers). One has to clearly separate the FDR concept from Benjamini-Hochberg's method. The first one is a generalized type-I error, while the second one is a multiple testing procedure which controls that error. This is very analogous for example to FWER and Bonferroni's procedure.

Indeed, there is no immediate reason why the number of hypotheses should matter when you want to use FDR controlling methods. It just depends on your goal. In particular, assume you are testing $m$ hypotheses and your procedure rejects $R$ of them with $V$ false rejections.

Now you use a FWER ($= \Pr[V \geq 1]$) controlling procedure if you want to make no type I errors. On the other hand, you use the $\text{FDR}$, when it is acceptable to make a few errors, as long as they are relatively few compared to all the rejections $R$ you made, i.e.

$$ \text{FDR} = \mathbb E\left[\frac{V}{\max{R,1}}\right]$$

Thus, the answer to your question completely depends on what you want to achieve and there is no intrinsic reason why small $m$ would be problematic. Just to illustrate a bit further: The data analysis example in Benjamini-Hochberg's seminal 1995 paper just included $m=15$ hypotheses, and of course it is also valid for that case!

Of course, there is a caveat to my answer: The BH procedure only got popular after "massive" (e.g. Microarrays) datasets started becoming available. And as you mention it is typically used for such "Big data" application. But this is just because in such cases the $\text{FDR}$ as a criterion makes more sense, e.g. because it is scalable and adaptive and facilitates exploratory research. The FWER on the other hand is very stringent, as required by clinical studies etc. and punishes you too much for exploring too many options simultaneously (i.e. not well suited to exploratory work).

Now, let's assume you have decided that the FDR is the appropriate criterion for your application. Is Benjamini Hochberg the right choice to control the FDR when the number of hypotheses is low? I would say yes, since it is statistically valid also for low $m$. But for low $m$ you could for example also use another procedure, namely Benjamini and Liu's procedure, which also controls the FDR. In fact, the authors suggest its use (over Benjamini-Hochberg) when $m \leq 14$ and most hypotheses are expected to be false. So you see that there are alternative choices for FDR control! In practice, I'd still use BH just because it is so well established and because the benefits of using Benjamini-Liu will be marginal in most cases if at all existent.

On a final related note, there are indeed some FDR controlling procedures which you should not use for low $m$! These include all local-fdr based procedures, for example as implemented in the R packages "fdrtool" and "locfdr".