Does false discovery rate depend on the p-value or only on the alpha level?

Question

Let's say I get a p-value of 0.001. I know that alpha level dictates the probability of a type I error, so if I get a result this significant, is my false discovery rate (FDR) lower than if I were to get a p-value of 0.05?

Assuming a power of 0.8, if I were to take 100 samples of drugs with 10 percent that actually work, then 8 will show up as significant, and with an alpha of .05, about 5 will be type I errors. Thus the FDR is 38% or 5/13.

However, if I happen to get a p-value super extreme like 0.001, will my FDR change? If so, why does it, considering that alpha and percent of working drugs seem to dictate the FDR, rather than the p-value of a sample.

Answer 1

The standard definition of false discovery rate (FDR) is the expected proportion of false discoveries among all discoveries. As you note and as explained here in other answers, it does not depend on the $p$-value but only on the $\alpha$.

I guess what you are thinking about, is the expected proportion of false discoveries among the discoveries with a given $p$-value, e.g. among the discoveries with $p=0.001$. Strictly speaking, this is not called "false discovery rate" anymore. However, it is a meaningful quantity.

To say it differently, false discovery rate is the probability of the null hypothesis being true given that $p\le\alpha$, i.e. $$\mathrm{FDR}=P(H_0 | p \le \alpha).$$ The quantity you seem to be thinking about is simply $P(H_0 | p)$. This latter quantity will of course depend on $p$, and will be lower for lower values of $p$.

David Colquhoun (2014) does call it "false discovery rate", see Section 10 of his paper An investigation of the false discovery rate and the misinterpretation of $p$-values, and uses simple simulation to show that the lower the $p$-value, the smaller this FDR-conditioned-on-the-p-value will be. I guess though that this terminology can be considered sloppy. Also, see here for some lengthy discussion of this paper.

Answer 2

There's a lot of terminology bouncing around here. Let's look at this table adapted from Sun et al 2006 in her introduction of stratified FDR.

+------------+------------------+---------------+--------+
|            | DECLARED-NON-SIG |  DECLARED SIG |  TOTAL |
+------------+------------------+---------------+--------+
|            |                  |               |        |
| Truth: H_0 |  U               |  V            |  M_0   |
|            |                  |               |        |
| Truth:H_1  |  T               |  S            |  M_1   |
|            |                  |               |        |
| Total      |  m-R             |  R            |  m     |
+------------+------------------+---------------+--------+

The FDR (False discovery rate) is defined to be $E[\frac{V}{R}]$. We know $R$, obviously, but we don't know $V$. We can never absolutely know $V$ in practice.

The value of $E[V]$ may be defined as, as you have noted, $m_0\alpha$, or $m \pi_0\alpha$. This latter expression says "the number of tests multiplied by the proportion of true nulls times alpha" where $\pi_0$ is the proportion of true nulls.

This expression can be expanded out to see that:

$FDR = E[\frac{V}{R}] = \frac{E[V]}{E[R]} = \frac{m \pi_0\alpha}{m \pi_0\alpha + m(1-\pi_0)(1-\beta(\alpha))} = \frac{ \pi_0\alpha}{\pi_0\alpha + (1-\pi_0)(1-\beta(\alpha)} = \frac{1}{1+(1/\pi_0-1)(1-\beta(\alpha))/ \alpha }$

That might look complex but it's really quite intuitive. The expected value of FDR is the number of tests which falsely reject simply based on chance ($m\pi_0\alpha$) divided by the $m\pi_0\alpha$ plus the number of true tests which are actually rejected ($m(1-\pi_0)$ = number of tests * proprotion not null, while $1-\beta(\alpha)$ corresponds to the proportion of true non-nulls detected at that alpha).

I hope that gives you a better understanding of what the word FDR means. Nowhere in there is the actual $P$ value of an individual test mentioned because FDR is a concept in repeated sampling; your $P$-value changes every time you redo a test, but your $\alpha$ level of acceptance must stay the same to estimate FDR.

A $P$-value of 0.001 may be unlikely under the null, but you're not rejecting all $P$-values $< 0.001$. If you set your $\alpha$ to 0.001 and rejected all tests under that P value, then yes, your false positive rate would be 0.001. However, that's not what you're doing.

If you want to estimate the false discovery rate at which your test would be on the cusp of significance you should look into Storey et. al 2003 and use $q$-values (lower-case). This defines each $p$-value in terms of false discovery rate, not false positive rate.

To wrap up, if you get a $P$-value of 0.001, does your FDR change? No. You can change your FDR by changing your $\alpha$, but when you are repeatedly sampling, a low $P$-value just means an unlikely null.

Answer 3

I think that you have to draw a finer distinction between properties of a statistical test (alpha, FDR) and properties of a particular result (p-value, Q)

The proportion you are considering (%38) is Q, not FDR. It describes one particular result. FDR is defined as the expected proportion of false positives among positives, FDR=E(Q). In other words, if you would repeat the experiment an infinite number of times, compute in each repetition the same proportion you've computed in your example and average across all repetitions, then you will have a single proportion that would be the FDR. From this description it's clear that the FDR cannot depend on any particular p-value (but it certainly depends on alpha).

I also suspect that you understand p-value and alpha as equivalents. They are not. Alpha is the probability of a false-positive, a property of the statistical test. A p-value is the probability of observing a particular result assuming its a negative. Hence if you get a small (or large) p-value, it does not say anything about the proportion of false positives. See this question for further elaboration of this point..