I came across this issue as I was analysing some genomic data sets looking for differential DNA methylation. I have generated p-values from the comparisons I am making, and am now looking into the various methods of multiple testing correction to see which method is most appropriate for my analysis (i.e. conservative vs. non-conservative, applicability of method).
My first attempt was to use the q-value approach of Storey, et al. in their 2003 PNAS paper.
However, upon reading the paper, and the associated R package vignette, it appears that data with a "U-shaped" p-value distribution do not satisfy the underlying assumptions of the q-value method.
According to the q-value R package vignette:
The “U-shaped” p-value histogram is a red flag. An important assumption behind the estimation performed in this package is that null p-values follow a Uniform(0,1) distribution, which would result in a p-value histogram where the right tail is fairly flat as in the Hedenfalk et al. p-values. U-shaped p-value histograms 5can indicate that a one-sided test was performed on data where there is signal in both directions, or it can indicate that there is dependence among the variables in the data.
I am curious to know which methods of multiple testing correction are applicable when the p-value distribution is not "uniformly distributed", and what would happen if I tried to use the q-value method on a U-shaped p-value distribution?
I have read that the Benjamini-Hochberg and BH-Yekutieli methods are applicable when there is "dependence" in the data, but I am wondering how a U-shaped p-val distribution would effect these tests also.
Any help is much appreciated!