When to care about FDR vs when to care about FWER

Question

When testing multiple hypotheses on the same set of collected data, we want of course to correct for the increased likelihood of false positives from multiple tests.

So far I've always used the Holm-Bonferroni method to adjust the Family-Wise Error Rate (FWER) to $\le 5\%$.

However, there's also the Benjamini-Hochberg method. From what I understand, it can be used to fix the expected False Discovery Rate (FDR) across all performed tests at (let's say) $\le 5\%$. It could thus allow a $> 5\%$ chance that at least one of our results is a false positive, but offers a much higher power as a compromise. In addition, I believe that it (like Holm-Bonferroni) doesn't require any assumptions about the the test themselves being dependent or independent. (Please correct me if any of that is false.)

I'm now wondering, from a very practical standpoint, when is it acceptable to use Benjamini-Hochberg to control FDR, and when must I strictly keep FWER $\le 5\%$?

For example, let's say we have an experiment and compare a number of interventions each against the baseline. They will be compared regarding several dependent variables. So we'd have a number of Hypotheses (e.g., $H_{ij}$: Variable $i$ differs significantly from the baseline in condition $j$). What factors now influence whether I should focus on FWER or FDR?

(If that helps, I'm in Human-Computer Interaction. As a comparatively young field, we unfortunately don't have strong statistics traditions or best practices to fall back on / they are not yet necessarily reliable)

Answer 1

In the simplest possible terms, you control FWER when you care about the result of the specific hypotheses tested whereas you control FDR when you care about the number of significant results.

Here's my take on an example justifying FDR: suppose, for instance, you are exploring an in vitro or pharmacodynamic study of the efficacy of a certain novel anti-neoplastic therapy. Suppose further this cancer under study is broadly characterized in terms of a number of qualitative and quantitative markers - such as metabolic rate, number and size of measurable lesions, qualitative status of non-target lesions, cancer antigen markers, gene expressions identified at baseline biopsies, etc. etc. etc. You sincerely believe that if the drug really kills the cancer, you expect all of these to change toward normal values. Of course, the disease may mutate, the assays may be false positives at baseline or follow-up, etc. etc. So I may look at the false discovery rate when inspecting each possible efficacy measure, even though the specific mechanism is unknown and not yet specified. More "hits" than expected under a non-associative status would suggest the drug activity is "promising".

Answer 2

Great question @Tobl, I had the same therefore I'm sharing the insights that I've been able to find hoping that they are enough or they can lead us to a constructive discussion.

I'm now wondering, from a very practical standpoint, when is it acceptable to use Benjamini-Hochberg to control FDR, and when must I strictly keep FWER ≤5%

As far as I know, it all depends on which is your context and what you're looking for: if you cannot have any false positive, then you choose to control the FWER, otherwise you usually go for the FDR control.

As an example, suppose you have to perform a medical trial for a very dangerous disease that is composed of multiple tests, and for which you are considered healthy if and only if all tests are positive. Having a false positive may lead to misjudging an ill patient as healthy, which potentially leads to the death of the patient, therefore your desiderata is not to have any false positive and therefore you control for the FWER. Side note: as far as I know, this is not how clinical tests are developed in real life, but I hope you got the intuition behind the example.

Let us now imagine a different scenario for which you are studying the behavior of a target variable $T$ in relation with other covariates $\mathbf{V}$ in a dataset (e.g. causal discovery scenarios). Suppose that you'd like to understand which are the most important variables on the dataset and you do that testing multiple hypotheses. You run your procedure correcting for the FWER but the tests are not powerful enough, therefore you don't get any meaningful result. Then, one way to proceed is to correct for the FDR.