6
$\begingroup$

When testing multiple hypotheses on the same set of collected data, we want of course to correct for the increased likelihood of false positives from multiple tests.

So far I've always used the Holm-Bonferroni method to adjust the Family-Wise Error Rate (FWER) to $\le 5\%$.

However, there's also the Benjamini-Hochberg method. From what I understand, it can be used to fix the expected False Discovery Rate (FDR) across all performed tests at (let's say) $\le 5\%$. It could thus allow a $> 5\%$ chance that at least one of our results is a false positive, but offers a much higher power as a compromise. In addition, I believe that it (like Holm-Bonferroni) doesn't require any assumptions about the the test themselves being dependent or independent. (Please correct me if any of that is false.)

I'm now wondering, from a very practical standpoint, when is it acceptable to use Benjamini-Hochberg to control FDR, and when must I strictly keep FWER $\le 5\%$?

For example, let's say we have an experiment and compare a number of interventions each against the baseline. They will be compared regarding several dependent variables. So we'd have a number of Hypotheses (e.g., $H_{ij}$: Variable $i$ differs significantly from the baseline in condition $j$). What factors now influence whether I should focus on FWER or FDR?

(If that helps, I'm in Human-Computer Interaction. As a comparatively young field, we unfortunately don't have strong statistics traditions or best practices to fall back on / they are not yet necessarily reliable)

$\endgroup$
8
  • 1
    $\begingroup$ Excellent question. My take would be that using FDR would go with more exploratory analyses earlier in a research program, and that FWER is more appropriate w/ more confirmatory analyses later in a research program. $\endgroup$ Mar 26, 2021 at 20:28
  • 1
    $\begingroup$ There is a certain positive dependence assumption that is needed to be assured of FDR control when using the B-H method. $\endgroup$ Mar 26, 2021 at 21:13
  • $\begingroup$ @BigBendRegion : Thanks for bringing that to my attention. I didn't know that negative dependence could endanger BH, though I also can't seem to think of an experiment design of mine in the last few years that featured negative dependence. $\endgroup$
    – Tobl
    Mar 28, 2021 at 22:05
  • 2
    $\begingroup$ I wouldn't go that far, but I'm not sure the FDA will go for FDR in new drug / device efficacy trials. Basically, if you are doing a prototypical experiment, the assumption is that you can get more data if you need it, and if a suite of specific follow on analyses is the point of the study, you power it for rigorous tests of those hypotheses. $\endgroup$ Mar 28, 2021 at 22:26
  • 1
    $\begingroup$ Putting this much emphasis on false positives and giving no consideration of false negatives is a major problem in my view. $\endgroup$ Dec 21, 2022 at 21:27

2 Answers 2

2
$\begingroup$

In the simplest possible terms, you control FWER when you care about the result of the specific hypotheses tested whereas you control FDR when you care about the number of significant results.

Here's my take on an example justifying FDR: suppose, for instance, you are exploring an in vitro or pharmacodynamic study of the efficacy of a certain novel anti-neoplastic therapy. Suppose further this cancer under study is broadly characterized in terms of a number of qualitative and quantitative markers - such as metabolic rate, number and size of measurable lesions, qualitative status of non-target lesions, cancer antigen markers, gene expressions identified at baseline biopsies, etc. etc. etc. You sincerely believe that if the drug really kills the cancer, you expect all of these to change toward normal values. Of course, the disease may mutate, the assays may be false positives at baseline or follow-up, etc. etc. So I may look at the false discovery rate when inspecting each possible efficacy measure, even though the specific mechanism is unknown and not yet specified. More "hits" than expected under a non-associative status would suggest the drug activity is "promising".

$\endgroup$
0
$\begingroup$

Great question @Tobl, I had the same therefore I'm sharing the insights that I've been able to find hoping that they are enough or they can lead us to a constructive discussion.

I'm now wondering, from a very practical standpoint, when is it acceptable to use Benjamini-Hochberg to control FDR, and when must I strictly keep FWER ≤5%

As far as I know, it all depends on which is your context and what you're looking for: if you cannot have any false positive, then you choose to control the FWER, otherwise you usually go for the FDR control.

As an example, suppose you have to perform a medical trial for a very dangerous disease that is composed of multiple tests, and for which you are considered healthy if and only if all tests are positive. Having a false positive may lead to misjudging an ill patient as healthy, which potentially leads to the death of the patient, therefore your desiderata is not to have any false positive and therefore you control for the FWER. Side note: as far as I know, this is not how clinical tests are developed in real life, but I hope you got the intuition behind the example.

Let us now imagine a different scenario for which you are studying the behavior of a target variable $T$ in relation with other covariates $\mathbf{V}$ in a dataset (e.g. causal discovery scenarios). Suppose that you'd like to understand which are the most important variables on the dataset and you do that testing multiple hypotheses. You run your procedure correcting for the FWER but the tests are not powerful enough, therefore you don't get any meaningful result. Then, one way to proceed is to correct for the FDR.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.