1
$\begingroup$

I have 11 tests and I expect at least half of them to be truly different (I expect that from prior knowledge, there are papers that directly indicate possible difference). But the power of each test is not that big, resulting p-values are not that small:

0.0403, 0.0726, 0.0289, 0.6864, 0.003, 0.3539, 0.3753, 0.1256, 0.0292, 0.0858, 0.0024

Do I really need a FDR correction in this situation? Can I say that it could be too strict, select all tests that gave a raw p-value less than 0.05 (I am working in the field where people still appreciate this magic 0.05 number) and describe the prior knowledge on why we expect that many positive results?

I am not aiming for a lot of positive results, I still have 2 "significant" after FDR correction, however, I feel that smth is wrong with applying FDR when I expect a lot of true positive results (but have small power unfortunately).

$\endgroup$
1

2 Answers 2

1
$\begingroup$

The false discovery rate (FDR), unlike familywise error rate methods, the FDR does not assume that all null hypotheses are true, but that:

  1. some null hypotheses may be false (i.e. true positives), and
  2. for each null hypothesis that is rejected sequentially, the estimated probability of the next null hypothesis in the sequence being a true positive is updated (increased).

Because you are testing multiple null hypotheses $\alpha$ cannot mean the probability of falsely rejecting a single null hypothesis as it can for a single hypothesis test. Therefore, you should adjust for multiple comparisons.

I would go farther, and say that you should conduct relevance tests (see, for example my answer), to explicitly parse out whether any negative results are because the differences are equivalent or trivially different, or simply because your test is under-powered. (You would still apply the FDR when conducting relevance tests.)

The $\text{H}^{-}_{0}$ has an a priori researcher-selected equivalence threshold which is interpreted as "the smallest effect large enough to matter."

$\endgroup$
2
  • $\begingroup$ Thank you a lot! The explanation is clear, so I will use corrected q-values for the report. $\endgroup$ Aug 7, 2018 at 7:16
  • 1
    $\begingroup$ @GermanDemidov You are welcome. Feel free to click the up arrow next to my answer to up-vote it. :) $\endgroup$
    – Alexis
    Aug 7, 2018 at 14:07
2
$\begingroup$

Another approach is to estimate the proportion of the tests for which the null hypothesis is true. For your data, it appears only about 30% (3 or 4 out of 11) of the null hypotheses are probably true, meaning that around 70% of the tests reflect truly different effects:

library(limma)
> p <- c(0.0403, 0.0726, 0.0289, 0.6864, 0.003,
+        0.3539, 0.3753, 0.1256, 0.0292, 0.0858, 0.0024)
> propTrueNull(p)
[1] 0.3005667

Furthermore, the 7 smallest p-values all have FDR values less than 14%

> sort(p.adjust(p, method="BH"))
[1] 0.01650 0.01650 0.08030 0.08030 0.08866 0.13310 0.13483
[8] 0.17270 0.41283 0.41283 0.68640

which implies that, out of the 7 smallest p-values, we expect no more than 1 of those null hypotheses to be true (since 1/7 = 14%).

$\endgroup$
2
  • 1
    $\begingroup$ Kjetil, I've gone ahead and restored the R code and output to standard R console format. $\endgroup$ Oct 22, 2022 at 7:54
  • $\begingroup$ Thanks for doing so! $\endgroup$ Oct 22, 2022 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.