6
$\begingroup$

One post says that we should check the distribution of p-value before apply FDR correction. If the p-value distribution doesn't behavior well(e.g. U shape, not uniformly distributed at the tail toward 1), there might be a problem of your data or your model assumption.

However, I'm confused by another post, saying that 'The FDR does not assume a uniform distribution of p-values'.

What's one should I follow? Can I use BH-FDR if the p-value is not uniformly distributed?

Thanks in advance!

Improve this question
$\endgroup$
8
  • $\begingroup$ Your first link says that uniform p-values are a bad thing. Your second link says FDR does not require uniform p-values, which makes sense because a uniform distribution implies there are no effects to find). Where is the conflict? $\endgroup$
    – John
    Commented Apr 4, 2016 at 23:34
  • $\begingroup$ The conflict is that whether I should do something else to correct the U-shape p-value distribution. $\endgroup$
    – user3915365
    Commented Apr 4, 2016 at 23:53
  • $\begingroup$ And that has nothing to do with a uniform p-distribution assumption which is what your question is currently focused on. I suggest you either extensively edit this question or delete it and open a new one that asks what you want to ask. Either way also say what you've done with respect to the recommendations in the first link for scenario C. Don't confuse the requirement that the small number of hypotheses close to 1 be uniform with a uniform distribution. $\endgroup$
    – John
    Commented Apr 4, 2016 at 23:57
  • $\begingroup$ When you were looking at a distribution of p-values, how did you know the null was true? $\endgroup$
    – Glen_b
    Commented Apr 5, 2016 at 0:11
  • $\begingroup$ @Glen_b We don't know whether null is true. But we expect the p-value distribution would follow the scenario A or B in the first post. $\endgroup$
    – user3915365
    Commented Apr 5, 2016 at 23:32

1 Answer 1

Reset to default
5
$\begingroup$

Benjamini Hochberg is valid as long as the null p-values are superuniform, this means:

$$ \Pr[P_i \leq t \mid H_0] \leq t $$

This is valid with "$=$" for uniform null p-values. It is also true for U-shaped mixture distributions (if the left peak of the U corresponds to alternatives, then a uniform component + a peak close to 1 will correspond to the null distribution, which consequently is subuniform). Also superuniformity holds for discrete distributions (which cannot lead to uniformly distributed p-values because of the discreteness).

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for this answer. Would you have any pointers to papers or websites that explain this in more detail? $\endgroup$
    – dlaehnemann
    Commented Jul 22, 2019 at 10:14
  • $\begingroup$ Just came across this other answer by @air that gives some more insight into how Benjamini-Hochberg FDR control works, and also provides some references I'll dig into: stats.stackexchange.com/a/178350/254369 $\endgroup$
    – dlaehnemann
    Commented Jul 22, 2019 at 11:34
  • 1
    $\begingroup$ I think what you define here, is actually "super-uniformity", not "sub-uniformity". I've dug into this a bit and there's a question where I provide a bunch of references for "super-uniformity" and "stochastical dominance over a standard uniform random variable" (stats.stackexchange.com/q/419005/254369). Should I edit your answer or would you prefer to do that? $\endgroup$
    – dlaehnemann
    Commented Jul 24, 2019 at 15:34
  • 1
    $\begingroup$ @dlaehnemann thank you excellent catch about superuniform. I fixed it now. I feel this result, that superuniformity suffices (instead of uniformity) for FDR control is almost a statistical "folklore" result. It just happens that for many FDR control methods the argument goes through unchanged with superuniform, sometimes having to replace an "=" with a "$\leq$" in a step of the proof, but this still works to bound the FDR. Sometimes the authors spell it out, sometimes not. $\endgroup$
    – air
    Commented Jul 26, 2019 at 5:39
  • 2
    $\begingroup$ One reference where the authors spell it out rigorously is "Two simple sufficient conditions for FDR control" by Roquain and Blanchard (EJS, 2008). See just a bit after their Definition 2.1. $\endgroup$
    – air
    Commented Jul 26, 2019 at 5:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.