Suppose that I perform two different comparisons on the same genomic dataset (with say 1000 genes, and three conditions).

For each comparison, I adjust the pvalues using Benjamini Hochberg, controlling the FDR. So, in each comparison, I would have say, 0.1 false positives on expectation.

Suppose that I do not control in any way for the fact that I perform two comparisons and not just one.

What would be the False Discovery Rate for the whole list of significantly changed genes for the two comparisons?

I think that if I set the threshold to be 0.1 then (for a single comparisons)

P(false positive) = 0.1

P(true positive) = 1 - 0.1

For two comparisons:

P(true positive) = (1 - 0.1)^2
P(false positive) = 1 - (1-0.1)^2

And thus the the FDR would be 1-(1-alpha)^n

But I would like some verification of that.


No, your calculations are not correct. You are assuming (1) that the Benjamini & Hochberg FDR adjusted p-value is equal to P(false positive) and (2) that the tests are statistically independent, neither of which are generally true. The mathematics of FDR control are very much more subtle and difficult than your calculations assume.

FDR behaves very differently to familywise type I error, which is the traditional target of older multiple testing adjustment methods. Unlike type I error, FDR is a scalable quantity. If you control the true FDR below 0.1 for two sets of hypotheses, then the true FDR is automatically below 0.1 for both sets combined. That follows from basic arithmetic.

The Benjamini & Hochberg expected FDR does not however obey such a scaling rule exactly. The limma package (which you are probably familiar with) is specifically designed for the sort of genomic analysis you are refering to. The limma::decideTests function implements several different methods for extending FDR control across several contrasts simultaneously. Apart from the "global" method, the results are approximate and there is no exact theory supporting these calculations.

Generally speaking, the scalability of exact FDR tends to mean that expected FDR also scales well in practice, so one does not need to worry very much about it. This rule of thumb works pretty well assuming the number of tests per contrast (i.e., the number of genes) is large compared to the number of contrasts being conducted, which is always true in the genomic context we are discussing (both here and in your many questions across the Bioconductor and Biostars help forums). If the number of contrasts was larger than the number of tests per contrast then one would need to apply FDR control globally.

  • $\begingroup$ I don't know if you have the time to refer me to a resource that would explain the issue, but the fact that expected FDR is scalable across multiple comparisons is completely counterintuitive for me. Each contrast is a statistical test, so how can performing multiple tests not change the expected FDR? By this logic, even if I am to perform a billion contrasts, I can still expect the same number of false positives as if I perform only one. $\endgroup$ – Sam Apr 11 at 9:03
  • 1
    $\begingroup$ @Sam FDR gives the rate of false positives, not the number of false positives. If you do lots of tests then the number of false positives will generally increase but rate will not necessarily do so. If I give you a billion bags of balls, and the proportion of red balls is less than 10% in every individual bag, then what is the proportion of red balls in all bags combined? $\endgroup$ – Gordon Smyth Apr 11 at 9:45
  • $\begingroup$ Does that mean that if I : perform deconvolution on bulk tissue; test DE genes for each cell sub-type separately; corect DE within each cell subtype using BH correction - then I do not need to worry about multiple comparisons problem due to multiple cell sub-types being tested? ( I have asked this here stats.stackexchange.com/questions/520608/… as a separate question) $\endgroup$ – Sam Apr 21 at 10:41
  • 1
    $\begingroup$ @Sam Yes, it would apply to different cell types the same as to different contrasts. $\endgroup$ – Gordon Smyth Apr 21 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.