5
$\begingroup$

Let's say I get a p-value of 0.001. I know that alpha level dictates the probability of a type I error, so if I get a result this significant, is my false discovery rate (FDR) lower than if I were to get a p-value of 0.05?

Assuming a power of 0.8, if I were to take 100 samples of drugs with 10 percent that actually work, then 8 will show up as significant, and with an alpha of .05, about 5 will be type I errors. Thus the FDR is 38% or 5/13.

However, if I happen to get a p-value super extreme like 0.001, will my FDR change? If so, why does it, considering that alpha and percent of working drugs seem to dictate the FDR, rather than the p-value of a sample.

$\endgroup$

3 Answers 3

2
$\begingroup$

The standard definition of false discovery rate (FDR) is the expected proportion of false discoveries among all discoveries. As you note and as explained here in other answers, it does not depend on the $p$-value but only on the $\alpha$.

I guess what you are thinking about, is the expected proportion of false discoveries among the discoveries with a given $p$-value, e.g. among the discoveries with $p=0.001$. Strictly speaking, this is not called "false discovery rate" anymore. However, it is a meaningful quantity.

To say it differently, false discovery rate is the probability of the null hypothesis being true given that $p\le\alpha$, i.e. $$\mathrm{FDR}=P(H_0 | p \le \alpha).$$ The quantity you seem to be thinking about is simply $P(H_0 | p)$. This latter quantity will of course depend on $p$, and will be lower for lower values of $p$.

David Colquhoun (2014) does call it "false discovery rate", see Section 10 of his paper An investigation of the false discovery rate and the misinterpretation of $p$-values, and uses simple simulation to show that the lower the $p$-value, the smaller this FDR-conditioned-on-the-p-value will be. I guess though that this terminology can be considered sloppy. Also, see here for some lengthy discussion of this paper.

$\endgroup$
2
  • 1
    $\begingroup$ Wow, great answer! Thanks! So I'm trying to understand: if I get a really small p-value, like 0.001, am I any more sure of the significance in comparison to getting a p-value of 0.045, where alpha is 0.05? In both cases, we're rejecting the null in favor of the alternative, but if our FDR is the same, can you even be more confident with a 0.001 p-value compared to a 0.045? $\endgroup$
    – rb612
    Aug 10, 2015 at 20:14
  • $\begingroup$ Good question. I would say yes. Note that in the Fisher's approach to statistical testing, p-value is taken as the strength of the evidence against the null; it is only the Neyman-Pearson approach that demands to choose an alpha beforehand and not pay attention to the p-value apart from checking if it below alpha or not. In general, this is one of the questions that tends to generate a lot of heated discussions. See e.g. Are smaller p-values more convincing? and other threads linked there (and there is a lot linked there). $\endgroup$
    – amoeba
    Aug 10, 2015 at 20:23
2
$\begingroup$

There's a lot of terminology bouncing around here. Let's look at this table adapted from Sun et al 2006 in her introduction of stratified FDR.

+------------+------------------+---------------+--------+
|            | DECLARED-NON-SIG |  DECLARED SIG |  TOTAL |
+------------+------------------+---------------+--------+
|            |                  |               |        |
| Truth: H_0 |  U               |  V            |  M_0   |
|            |                  |               |        |
| Truth:H_1  |  T               |  S            |  M_1   |
|            |                  |               |        |
| Total      |  m-R             |  R            |  m     |
+------------+------------------+---------------+--------+

The FDR (False discovery rate) is defined to be $E[\frac{V}{R}]$. We know $R$, obviously, but we don't know $V$. We can never absolutely know $V$ in practice.

The value of $E[V]$ may be defined as, as you have noted, $m_0\alpha$, or $m \pi_0\alpha$. This latter expression says "the number of tests multiplied by the proportion of true nulls times alpha" where $\pi_0$ is the proportion of true nulls.

This expression can be expanded out to see that:

$FDR = E[\frac{V}{R}] = \frac{E[V]}{E[R]} = \frac{m \pi_0\alpha}{m \pi_0\alpha + m(1-\pi_0)(1-\beta(\alpha))} = \frac{ \pi_0\alpha}{\pi_0\alpha + (1-\pi_0)(1-\beta(\alpha)} = \frac{1}{1+(1/\pi_0-1)(1-\beta(\alpha))/ \alpha }$

That might look complex but it's really quite intuitive. The expected value of FDR is the number of tests which falsely reject simply based on chance ($m\pi_0\alpha$) divided by the $m\pi_0\alpha$ plus the number of true tests which are actually rejected ($m(1-\pi_0)$ = number of tests * proprotion not null, while $1-\beta(\alpha)$ corresponds to the proportion of true non-nulls detected at that alpha).

I hope that gives you a better understanding of what the word FDR means. Nowhere in there is the actual $P$ value of an individual test mentioned because FDR is a concept in repeated sampling; your $P$-value changes every time you redo a test, but your $\alpha$ level of acceptance must stay the same to estimate FDR.

A $P$-value of 0.001 may be unlikely under the null, but you're not rejecting all $P$-values $< 0.001$. If you set your $\alpha$ to 0.001 and rejected all tests under that P value, then yes, your false positive rate would be 0.001. However, that's not what you're doing.

If you want to estimate the false discovery rate at which your test would be on the cusp of significance you should look into Storey et. al 2003 and use $q$-values (lower-case). This defines each $p$-value in terms of false discovery rate, not false positive rate.

To wrap up, if you get a $P$-value of 0.001, does your FDR change? No. You can change your FDR by changing your $\alpha$, but when you are repeatedly sampling, a low $P$-value just means an unlikely null.

$\endgroup$
8
  • 2
    $\begingroup$ I don't claim to be an expert in FDR's, but I'm very curious about the equation $E[V/R] = E[V] / E[R]$ you have written. Obviously $E[X/Y] = E[X] / E[Y]$ is not true in general (even if $X$ and $Y$ are independent). $\endgroup$
    – guy
    Aug 10, 2015 at 14:48
  • $\begingroup$ @guy, I claim no explicit knowledge on this subject, but in the Sun paper, she notes that the $E[V/R]=E[V]/E[R]$ was proven in the 2003 Storey paper. If you follow the link in the answer, his discussion on the subject is in the methods section. $\endgroup$
    – Chris C
    Aug 10, 2015 at 14:57
  • $\begingroup$ @guy Yes you are right, I have no idea how this made it into the paper without any reviewer noticing. On the other hand, under some assumptions (for example under independence of all p-values), the equality holds up to a factor $\mathcal{O}\left(\frac{1}{\sqrt{m}}\right)$. $\endgroup$
    – air
    Oct 22, 2015 at 16:58
  • 1
    $\begingroup$ Part 1) I mean that @guy is correct, in the sense that the equality $\text{FDR} = \frac{E[V]}{E[R]}$ is wrong. An easy way to see this is the case when $m_0=m$ ($\pi_0=1$). Then the RHS is $1$ (since every rejection is a false rejection), but the LHS is $\text{FDR}=\text{FWER} = \Pr[ R \geq 1] < 1$ , with any reasonable method that controls the $\text{FDR}$. To see this recall that in the definition of $\text{FDR}$, the ratio $\frac{V}{R}$ is set to $0$ when nothing is rejected. $\endgroup$
    – air
    Oct 22, 2015 at 17:45
  • 1
    $\begingroup$ Part 2) For the other part of my statement, I meant that under certain technical conditions (e.g. under independence of the p-values and $\pi_0<1$; I forgot to mention the second condition in my initial comment), asymptotically the equality holds: $E[\frac{V}{R}] = \frac{E[V]}{E[R]} + \mathcal{O}\left(\frac{1}{\sqrt{m}}\right)$. Thus if there is at least some signal, then this equality holds asymptotically (for increasing number of tests $m$). This (asymptotic) equality is mentioned in normaldeviate.wordpress.com/2012/10/04/… (check the comments too). $\endgroup$
    – air
    Oct 22, 2015 at 17:52
1
$\begingroup$

I think that you have to draw a finer distinction between properties of a statistical test (alpha, FDR) and properties of a particular result (p-value, Q)

The proportion you are considering (%38) is Q, not FDR. It describes one particular result. FDR is defined as the expected proportion of false positives among positives, FDR=E(Q). In other words, if you would repeat the experiment an infinite number of times, compute in each repetition the same proportion you've computed in your example and average across all repetitions, then you will have a single proportion that would be the FDR. From this description it's clear that the FDR cannot depend on any particular p-value (but it certainly depends on alpha).

I also suspect that you understand p-value and alpha as equivalents. They are not. Alpha is the probability of a false-positive, a property of the statistical test. A p-value is the probability of observing a particular result assuming its a negative. Hence if you get a small (or large) p-value, it does not say anything about the proportion of false positives. See this question for further elaboration of this point..

$\endgroup$
4
  • $\begingroup$ But can't one condition FDR on a particular p-value, e.g. ask what is $E(Q | p=0.001 \pm 0.0005)$? This will presumably be much lower than $E(Q | p=0.04 \pm 0.0005)$ And your definition of FDR is essentially $E(Q)=E(Q | p<0.05)$, right? $\endgroup$
    – amoeba
    Aug 10, 2015 at 9:16
  • $\begingroup$ The definition I used does not condition Q on p. It is conditioned on alpha. And why would E(Q|p=0.001) be smaller than E(Q|p=0.04)? $\endgroup$ Aug 10, 2015 at 10:15
  • $\begingroup$ Consider the example given by the OP. Imagine 1 million experiments, with 90% true nulls and 10% with true effect, and power 80%. Subselect all experiments with a given p-value and compute Q in this subselection. Then for smaller p-values it will be lower. If you subselect all experiments with p<alpha, then you get your standard Q. What I mean, is that Q is defined as proportion of false discoveries among the discoveries, but we can also ask about the proportion of false discoveries among the discoveries with a given p-value. I think this is what OP's question is about. $\endgroup$
    – amoeba
    Aug 10, 2015 at 10:34
  • $\begingroup$ The OP asked 'If so, why does it, considering that alpha and percent of working drugs seem to dictate the FDR, rather than the p-value of a sample. ' from that I conclude he's interested in the standard FDR and not in a conditioned Q. Regarding your simulation, How many comparisons do have within each experiment? $\endgroup$ Aug 10, 2015 at 10:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.