I have a list of p-values from a series of regression equations. Many of these P-values are small. To compute false discovery rate I have been using the q-value package in R. I notice, based on this analysis, that P-values of 0.05 correspond to q-values that are smaller than the p values (0.03). Would I then say that my false discovery rate is smaller than my P value? Is this possible?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
Yes, this is possible, if the proportion of null hypotheses (which is estimated by the qvalue package based on your p-value distribution) is small and your test is powerful.
Here's an example. Let's say you're testing 1000 hypotheses, and let's say 200 (20%) are actually null- this proportion is called $\pi_0$. Assume the qvalue package accurately estimates this value (see here for more on how it does that, and note that you can see what it estimates for your data with qvalue(mypvalues)$pi0). Furthermore, let's say that 500 of your p-values are under .05 (your test is powerful). Then what would be the q-value corresponding to a p-value of .05?
The q-value is the expected proportion of false discoveries you would obtain by setting a particular p-value cutoff. In your 200 null hypotheses, the p-values are uniformly distributed between 0 and 1 (that's part of the definition of a p-value). This means that 5% of them (10 hypotheses) will fall under .05. So you have 500 hypotheses under .05, and you expect 10 of them to be false: your expected FDR is $10/500=.02%$. In this case, the q-value is indeed smaller than the p-value.
answered May 7, 2014 at 15:23
-
$\begingroup$ I have two datasets, one with the issue above (pvals > qvals) and a dataset that followed the expected (pvals < qvals). Following your post, however, i noticed that the latter dataset has a π0 of 1.0. Can I then trust the hypotheses that fall below a certain FDR (0.05)? $\endgroup$ Commented Jun 11, 2020 at 23:07
-
1$\begingroup$ @AlanGarciaElfring generally yes! Most methods to estimate $\pi_0$ are conservative (they intentionally prefer an overestimate to an underestimate), which means it could be estimated at 1 even if there are true positives. When $\pi_0$ is 1, then the q-value method becomes equivalent to Benjamini-Hochberg FDR control. $\endgroup$ Commented Jun 12, 2020 at 3:34
$\begingroup$
$\endgroup$
You do not compute the false discovery rate, you control the false discovery rate.
Family-wise error rate methods (e.g. Bonferroni, Holm-Sidák, etc.) attempt to control the probability of making a false rejection of H$_{0}$ while assuming that all null hypotheses are true.
False discovery rate methods attempt to control the probability of of making a false rejection of H$_{0}$ while assuming that some null hypotheses are false. In this context $q$-values, are $p$-values that have been adjusted using a method to control the false discovery rate. $q$-values should be greater than or equal to the $p$-values from which they are computed, but it is difficult to say more about why you are getting the results you are without seeing your code.
