Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

I am trying to program a pathway analysis. Briefly, I have a list of genes which contain 1 or more mutations. I have mapped these genes to pathways, and I want find out if any pathways are over-represented in these mappings.

My approach is permutation based: I randomly place an equivalent number of mutations within the genome (taking gene length) into account, and then I do this 10,000 times, and count up the number of times each pathway got a certain number of mutations. Plotting a histogram of these permutations for any one pathway looks like a chi sq distribution plotted in R with a df=1 and ncp=0.

enter image description here

My question is how do I now calculate a p-value based on this distribution? Any help is appreciated.

share|improve this question

1 Answer

up vote 1 down vote accepted

If this represents the distribution of your test statistic under the null hypothesis and you want a one-sided level 0.05 test find the 95th percentile of this empirical distribution. That will be your critical value. Compare it to the observed value of the test statistic based on the original sample. If it exceeds the critical value you can reject the null hypothesis, otherwise you can not. If you want the p-value look at the value of your test statistic and count the proportion of values from your empirical null distribution that are equal to or greater than your observed value of the test statistic.

share|improve this answer
Thanks. Excuse my extreme ignorance, but by find the 95th percentile do you mean find the point above which the observations are in the top 5% of the distribution? This might be difficult since the range of values I'm dealing with can potentially be small, and the observed test statistic is higher than any observed value in the empirical distribution. Should I simply run more permutations? – Davy Kavanagh Jul 25 '12 at 20:54
Yes that is what it means. For the critical value find the first percentile of the distribution that exceeds 95% and use that as your critical value. Of course you could generate more permutations to get a finer representation of the null distribution and if that is easy to do then do it. In any case there still may not be an exact 95th percentile. So you still take the first one above 95% for the critical value. If the highest value from the permutation distribution is above the observed value of the test statistic determining the p-value is straighforward. – Michael Chernick Jul 25 '12 at 21:13
If it isn't then suppose you did n permutations. I would simply say the p-value is less than 1/n or 1/(n+1) (whichever you prefer). – Michael Chernick Jul 25 '12 at 21:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.