# How to test if a subset of point is localized in the tail of distribution

I've a bunch of genes (~8000). Each of these genes has an associated p-value representing its "importance" in a specific biological pathway. Within these genes I'm interested in a subset of genes (~200). How can I test wheter these 200 genes are in the tail of the p-value distribution. In other terms, how can I test wheter these 200 genes are "more" significant than the rest of the genes. For now I did an hypergeometric test using a thresold on the p-value. Let's say :

A : Total genes (~8000)
B : Subset of gene of interest (~200)
C : Genes in A with p-values <= 0.05
D : Genes in B with p-values <= 0.05


and then

phyper(D,C,A-C,B)


but I have to define a thresold (0.05) ....

Edit : Can I do a Mann-Whitney U test on the p-values ? (comparing A vs B ; by removing the subset B from A of course)

Thanks

• I think this is called GSEA (gene set enrichment analysis). The broad institute has a tool for just running this called gsea and there are many R packages on bioconductor that can do it as well – ashokragavendran Jun 9 '16 at 9:33
• Yes I know they use a hypergeometric-like test to do that. Thanks ;) – Nicolas Rosewick Jun 9 '16 at 9:46
• I think they also use some sort of random walk approach for a non parametric test – ashokragavendran Jun 9 '16 at 9:51

Different possibilities come to mind.

• You could run a two-sample Kolmogorov-Smirnov test to check whether both p value distributions come from the same population. This would also come out significant of the smaller set is systematically larger than the rest, though.

• You could do a simple one-sided two-sample unpaired t test (without assuming equal variance) to check whether one sample's mean is lower than the other.

• If you want to test against specific violations of uniformity, you could use Neyman's smooth test - the different components "look at" different sections of the unit interval. Ledwina (1994, JASA) offers a way to select the number of components based on BIC - it's implemented in the ddst package for R.

• Plotting the empirical cumulative distribution function of both samples against each other would certainly be informative (this is how the Kolmogorov-Smirnov test works) - it wouldn't be a test, though. I am a bit unsure about how valid it is to test the distributions of p values against each other the way you propose, so I personally would trust such an ECDF plot more than some test, anyway.

• Thanks. Can I do a t-test on p-values ? I was thinking at a Mann-Whitney that check the ranking of the p-values across the two groups. What do you think ? – Nicolas Rosewick Jun 9 '16 at 8:21
• You can certainly run a t-test to check whether means are significantly different, or a Mann-Whitney. With your large samples, it likely won't make a difference. (Note that data don't need to be normally distributed for the t-test to be valid - we need the normal distribution of the means, and with $n=200$, we are "asymptotical enough". Run both tests; I'd expect the p values to be very similar.) – Stephan Kolassa Jun 9 '16 at 8:24
• Mann-Whitney : 9.366417e-18 T-test : 1.789273e-12 – Nicolas Rosewick Jun 9 '16 at 8:42
• That is identical for all intents and purposes. Remember that p values also exhibit sampling variability and cannot meaningfully be calculated to arbitrary precision (Boos & Stefanski, 2011, The American Statistician). – Stephan Kolassa Jun 9 '16 at 8:48
• Ok I think I will take a one-tailed Mann-Whitney test/ Thank you for your help and remarks. – Nicolas Rosewick Jun 9 '16 at 9:01