2
$\begingroup$

I have a quick question about using the Benjamini-Hochberg procedure to control FDR using the p.adjust() function in R. I have read that in order to properly apply an FDR control, your p-value distribution should look something like it does in "Histogram of pLeft". This is, as I understand it, because this and most other algorithms have as an assumption that your null p-values are uniformly distributed.

enter image description here

However, the p-values that I obtained from our experiment (identifying significantly enriched GO Categories between two tissue types) conform to a distribution like the one seen in "Histogram of pBimodal". This is not surprising to us, but does present a problem in that we want to be able to control the FDR given that we have ~3000 independent tests (even more for a second dataset with a similar distribution).

enter image description here

My primary question is "Is there a way to apply a correction given the p-value distribution seen in 'Histogram of pBimodal'"? I've heard that one can apply Benjamini-Hochberg to a p-value vector with a bimodal distribution, but that it will increase the Type-II error rate compared to what it would be were the data unimodal. I've also tested this using simulated data in R, and it appears to hold generally true when the number of hypotheses tested is held constant. I'm much more concerned with the Type-I error rate than the Type-II error rate for our data, if it is a valid approach. That is, I'd rather exclude some truly enriched GO categories than include a number of non-enriched categories. If not, could anyone recommend a different method to go about this?

$\endgroup$
  • $\begingroup$ Are you aware of this link: varianceexplained.org/statistics/interpreting-pvalue-histogram ? $\endgroup$ – vkehayas May 4 '18 at 10:46
  • $\begingroup$ I am, that is how I discovered that this is was an issue. However, their suggestions unfortunately don't apply well to this case. I've discovered that the only risk of applying this procedure to my data as it stands is that π<sub>0</sub> will increase, making the correction more conservative. I'm willing to accept the risk of a few more Type-II errors. I'd rather exclude a couple of false-negatives than include a bunch of false-positives. Edit: the "π<sub>0</sub>" should read "π naught". Not sure why that didn't subscript. I guess it just doesn't in comments? $\endgroup$ – Adam Rork May 4 '18 at 15:58
0
$\begingroup$

Actually, I suppose that another valid approach would be just lowering our α threshold to 0.01 instead of 0.05. That way the number of expected false discoveries would be around or fewer than 30 for 3000 tests rather than 150, which seems much more reasonable.

I may actually just go that route, but I'd still be interested in knowing peoples' thoughts regards my above post!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.