2
$\begingroup$

If we are have calculated multiple P values from independent tests, we can correct for the False Discovery Rate by using:

Ps <- c(0.04,0.02,0.8,0.9,0.004)
stats::p.adjust(Ps, method = "BH", n = length(Ps))

Tied p-values are adjusted in the same way:

Ps <- rep(c(0.0001,0.02,0.8,0.9,0.004), 2)
stats::p.adjust(Ps, method = "BH", n = length(Ps))

If our p values are calculated as

P = 2*((1 + sum(observed_efects <= null_effects))/(N+1))

Where N is the number of samples in the null distribution, our minimum P value is limited as

min(P) == 1/(N+1)

If we have a collection of P values in which all P values are equal to 1/(N+1) then is the FDR appropriate given all P values are given the same rank? e.g.

Ps <- rep(0.002, 555)
stats::p.adjust(Ps, method = "BH", n = length(Ps))
$\endgroup$
2
  • $\begingroup$ Yes, the FDR calculation works for any set of independent p-values, as you have already said yourself. It would be easier to give a more elaborate answer if you explained why you think there might be problem. There is no problem in BH's theorem or in the p.adjust calculation with tied p-values. The result given by p.adjust in this case seems perfectly natural --- it says we that expect no more than 1 false discovery amongst these 555 tests. $\endgroup$ Commented Aug 5, 2022 at 10:36
  • $\begingroup$ Ignoring the false negative probability may mean that the results are unreliable. $\endgroup$ Commented Aug 9, 2022 at 12:21

1 Answer 1

5
+50
$\begingroup$

The Benjamini-Hochberg FDR calculation performed by p.adjust seems perfectly appropriate in your situation. There is no theoretical problem with tied p-values in Benjamini and Hochberg's 1995 theorem on which the p.adjust code is based.

For your data, p.adjust returns a FDR value of 0.002 for all tests. Since you did 555 tests and $0.002\times 555=1.11$, this says that you expect no more than 1 false discovery if you reject all 555 null hypotheses. This seems to me to be a very reasonable and useful result. It agrees with intuition because clearly you could not have not have got all the p-values at the lowest possible value unless virtually all the null hypotheses were false.

The BH algorithms works on an ordered list of p-values and each FDR value applies to all the tests with p-values less than or equal to the current value. If all the p-values are equal, then all the FDR values will be equal and will apply to the entire ensemble of tests.

The BH algorithm works in such a way that the expected FDR for the whole set of tests is equal to the largest p-value. If all the p-values are equal, as they are in your case, then only the overall FDR is relevant.

By contrast, FWER multiple testing adjustments would not work so well. Both the Bonferroni and Holm multiple testing adjustments would give no significance at all, with all adjusted p-values equal to 1.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.