I've a bunch of genes (~8000). Each of these genes has an associated p-value representing its "importance" in a specific biological pathway. Within these genes I'm interested in a subset of genes (~200). How can I test wheter these 200 genes are in the tail of the p-value distribution. In other terms, how can I test wheter these 200 genes are "more" significant than the rest of the genes. For now I did an hypergeometric test using a thresold on the p-value. Let's say :

A : Total genes (~8000)
B : Subset of gene of interest (~200)
C : Genes in A with p-values <= 0.05
D : Genes in B with p-values <= 0.05 

and then


but I have to define a thresold (0.05) ....

Edit : Can I do a Mann-Whitney U test on the p-values ? (comparing A vs B ; by removing the subset B from A of course)


  • $\begingroup$ I think this is called GSEA (gene set enrichment analysis). The broad institute has a tool for just running this called gsea and there are many R packages on bioconductor that can do it as well $\endgroup$ – ashokragavendran Jun 9 '16 at 9:33
  • $\begingroup$ Yes I know they use a hypergeometric-like test to do that. Thanks ;) $\endgroup$ – Nicolas Rosewick Jun 9 '16 at 9:46
  • $\begingroup$ I think they also use some sort of random walk approach for a non parametric test $\endgroup$ – ashokragavendran Jun 9 '16 at 9:51

Different possibilities come to mind.

  • You could run a two-sample Kolmogorov-Smirnov test to check whether both p value distributions come from the same population. This would also come out significant of the smaller set is systematically larger than the rest, though.

  • You could do a simple one-sided two-sample unpaired t test (without assuming equal variance) to check whether one sample's mean is lower than the other.

  • If you want to test against specific violations of uniformity, you could use Neyman's smooth test - the different components "look at" different sections of the unit interval. Ledwina (1994, JASA) offers a way to select the number of components based on BIC - it's implemented in the ddst package for R.

  • Plotting the empirical cumulative distribution function of both samples against each other would certainly be informative (this is how the Kolmogorov-Smirnov test works) - it wouldn't be a test, though. I am a bit unsure about how valid it is to test the distributions of p values against each other the way you propose, so I personally would trust such an ECDF plot more than some test, anyway.

  • $\begingroup$ Thanks. Can I do a t-test on p-values ? I was thinking at a Mann-Whitney that check the ranking of the p-values across the two groups. What do you think ? $\endgroup$ – Nicolas Rosewick Jun 9 '16 at 8:21
  • $\begingroup$ You can certainly run a t-test to check whether means are significantly different, or a Mann-Whitney. With your large samples, it likely won't make a difference. (Note that data don't need to be normally distributed for the t-test to be valid - we need the normal distribution of the means, and with $n=200$, we are "asymptotical enough". Run both tests; I'd expect the p values to be very similar.) $\endgroup$ – Stephan Kolassa Jun 9 '16 at 8:24
  • $\begingroup$ Mann-Whitney : 9.366417e-18 T-test : 1.789273e-12 $\endgroup$ – Nicolas Rosewick Jun 9 '16 at 8:42
  • $\begingroup$ That is identical for all intents and purposes. Remember that p values also exhibit sampling variability and cannot meaningfully be calculated to arbitrary precision (Boos & Stefanski, 2011, The American Statistician). $\endgroup$ – Stephan Kolassa Jun 9 '16 at 8:48
  • $\begingroup$ Ok I think I will take a one-tailed Mann-Whitney test/ Thank you for your help and remarks. $\endgroup$ – Nicolas Rosewick Jun 9 '16 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.