Sliding threshold for hypergeometric test

Question

I have a list of 20,000 genes with an associated p-value (the total number of balls in the urn).

I also have a list of specific genes that are "targets" of interest (these are the white balls in the urn).

I want to do a hypergeometric test to see if there's an over-representation of targets among the genes I call significant (i.e., overabundance of white balls from the balls I draw from the urn).

Easy so far with phyper in R. But I can change the number of significant genes (number of balls I draw) by varying the p-value threshold I'm using to call something significant (.001, .01, .05, etc).

What I'm wondering - is there an alternative test that examines an overabundance of "targets" (white balls) at the top of the list of genes ranked by p-value? That is, can I do this test without specifying a p-value cutoff? That is, something like a more generalized hypergeometric test where I don't have to specify the number of balls I draw, that computes a result for all possible number of balls drawn (genes selected)?

Thanks.

Answer 1

I don't know GWAS, but it seems to me you could just test the two entire p-value distributions (i.e., target vs. non-target) directly via something like the Anderson-Darling test or plot them with a qq-plot. You could also model the two p-value distributions (i.e., target vs. non-target) as two different beta distributions using MLE. Alternatively, you could use beta regression with target status as a dummy. In any case, the idea would simply be that the distribution of target p-values is shifted down relative to the non-targets.

If it helps, I have a simple R example using beta regression here: Remove effect of a factor on continuous proportion data using regression in R.

Answer 2

I think this is quite a common question and there is no exact statistical test (that I know of) for it. Instead, you can do something like gene set enrichment analysis (GSEA) for the ranked list of genes (ranked by the p-values).

Answer 3

Sounds like gene set enrichment analysis (GSEA) would be one option. GSEA will rank genes by a metric (P-values) and then look for 'enrichment' of target genes at the top of the list. Does so by calculating an "enrichment score". Here is an excerpt from Subramaniana et al. PNAS 2005 "The score is calculated by walking down the list L [total # of balls in urn], increasing a running-sum statistic when we encounter a gene in S [white balls in urn] and decreasing it when we encounter genes not in S [non-white balls in urn]. The magnitude of the increment depends on the correlation of the gene with the phenotype [in your case P-value]. The enrichment score is the maximum deviation from zero encountered in the random walk; it corresponds to a weighted Kolmogorov–Smirnov-like statistic." Permutations are used to derive an empirical significance level. There is R code here: http://www.broadinstitute.org/cancer/software/gsea/wiki/index.php/R-GSEA_Readme. Its not maintained but I recently used it and found that it worked well.