5
$\begingroup$

I have learned many models and I calculated p-values for the cross-validation errors. I want to select significant models based on the false discovery rate (FDR). How can I estimate the FDR from p-value distribution?

$\endgroup$
2
  • $\begingroup$ I strongly recommend looking at Andrew Gelman’s writings on this. Most statisticians would argue that the true false discovery rate is always 0%, because the null hypothesis is always false. $\endgroup$ Mar 20, 2022 at 0:49
  • $\begingroup$ Why are you trying to estimate the false discovery rate? What’s your goal in your study? Are you trying to adjust for multiple comparisons? $\endgroup$ Mar 20, 2022 at 1:01

1 Answer 1

-1
$\begingroup$

Generally speaking, the false discovery rate correction requires ranking your p-values and then computing a critical value (based on the false discovery rate) that you can compare to your p-values. This website gives a very concrete example: here.

If you're using R, then it's even easier to do the adjustment. The following code is a quick example:

pvals <- c(0.04, 0.03, 0.06, 0.01, 0.02, 0.003)
p.adjust(p = pvals, method = "BH")

In the above code, the object pvals is a list of the p-values that you obtained from your analyses. The function p.adjust provides several different options for adjusting p-values based on multiple observations. In this case, the false discovery rate adjustment can be done by specifying method = "BH" for Benjamin-Hochberg or method = "fdr" for false discovery rate (note: they are the exact same procedure and will give the exact same results).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.