I came across this issue as I was analysing some genomic data sets looking for differential DNA methylation. I have generated p-values from the comparisons I am making, and am now looking into the various methods of multiple testing correction to see which method is most appropriate for my analysis (i.e. conservative vs. non-conservative, applicability of method).

My first attempt was to use the q-value approach of Storey, et al. in their 2003 PNAS paper.

However, upon reading the paper, and the associated R package vignette, it appears that data with a "U-shaped" p-value distribution do not satisfy the underlying assumptions of the q-value method.

According to the q-value R package vignette:

The “U-shaped” p-value histogram is a red flag. An important assumption behind the estimation performed in this package is that null p-values follow a Uniform(0,1) distribution, which would result in a p-value histogram where the right tail is fairly flat as in the Hedenfalk et al. p-values. U-shaped p-value histograms 5can indicate that a one-sided test was performed on data where there is signal in both directions, or it can indicate that there is dependence among the variables in the data.

enter image description here

My data happen to have a somewhat "U-shaped" distribution of p-values, so I guess this means q-value is not applicable?
enter image description here

I am curious to know which methods of multiple testing correction are applicable when the p-value distribution is not "uniformly distributed", and what would happen if I tried to use the q-value method on a U-shaped p-value distribution?

I have read that the Benjamini-Hochberg and BH-Yekutieli methods are applicable when there is "dependence" in the data, but I am wondering how a U-shaped p-val distribution would effect these tests also.

Any help is much appreciated!

  • $\begingroup$ Were your p-values calculated using 1-tailed tests? $\endgroup$ – gung - Reinstate Monica Nov 23 '15 at 19:53
  • $\begingroup$ I would think they were calculated using 2-tailed tests, but I'm having trouble finding that info in the paper. The method I used says it uses a modified likelihood ratio test. $\endgroup$ – Reilstein Nov 23 '15 at 20:00
  • $\begingroup$ The paper describing the statistical method states that the test is: "Motivated by the two-sample t-test, which can be considered a likelihood ratio test conditional on the estimated variance, we propose the approximation , where tp is the Student t distribution with p degrees of freedom." $\endgroup$ – Reilstein Nov 23 '15 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.