12
$\begingroup$

I am trying to apply Fisher's exact test in a simulated genetics problem, but the p-values appear to be skewed to the right. Being a biologist, I guess I'm just missing something obvious to every statistician, so I would greatly appreciate your help.

My setup is this: (setup 1, marginals not fixed)
Two samples of 0s and 1s are randomly generated in R. Each sample n = 500, probabilities of sampling 0 and 1 are equal. I then compare the proportions of 0/1 in each sample with Fisher's exact test (just fisher.test; also tried other software with similar results). Sampling and testing is repeated 30 000 times. The resulting p-values are distributed like this: p-value distribution

Mean of all the p-values is around 0.55, 5th percentile at 0.0577. Even the distribution appears discontinuous on the right side.

I've been reading everything I can, but I don't find any indication that this behavior is normal - on the other hand, it's just simulated data, so I see no sources for any bias. Is there any adjustment I missed? Too small sample sizes? Or maybe it's not supposed to be uniformly distributed, and the p-values are interpreted differently?
Or should I just repeat this a million times, find the 0.05 quantile, and use that as the significance cutoff when I apply this to actual data?

Thanks!

Update:

Michael M suggested fixing the marginal values of 0 and 1. Now the p-values give a much nicer distribution - unfortunately, it's not uniform, nor of any other shape I recognize:

p-vals w fixed marginals

adding the actual R code: (setup 2, marginals fixed) 

samples=c(rep(1,500),rep(2,500))
alleles=c(rep(0,500),rep(1,500))
p=NULL
for(i in 1:30000){
  alleles=sample(alleles)
  p[i]=fisher.test(samples,alleles)$p.value
}
hist(p,breaks=50,col="grey",xlab="p-values",main="")

Final edit:
As whuber points out in the comments, the areas just look distorted due to binning. I am attaching the QQ-plots for the setup 1 (free marginals) and setup 2 (fixed marginals). Similar plots are seen in Glen's simulations below, and all these results in fact seem rather uniform. Thanks for the help!

pval-qqplot

Improve this question
$\endgroup$
11
  • 2
    $\begingroup$ Try to repeat your simulation while holding not only the group sizes (500 each) but also the sum of "1" (over the pooled sample) constant. The p value of Fisher's exact test is derived under this "fixed marginal distribution" setting. Does the picture look better then? Btw. you cannot expect the p-value distribution to be exactly uniform by the discrete nature of the sampling distribution (i.e. the hypergeometric). $\endgroup$
    – Michael M
    Commented Jul 21, 2015 at 19:57
  • 1
    $\begingroup$ It might be useful to have look at your R code. $\endgroup$
    – conjugateprior
    Commented Jul 21, 2015 at 20:47
  • 1
    $\begingroup$ @Glen it seems to me from the code that in each iteration both samples have the same number of 0s and 1s (i.e. null hypothesis should hold) or am I wrong? $\endgroup$
    – bdeonovic
    Commented Jul 21, 2015 at 22:55
  • 5
    $\begingroup$ These histograms look remarkably uniform to me. You have to remember that histograms display probability (or frequency) by means of area. The increasing gaps to the right (due to the unavoidable discreteness of the p-value distribution of any nonrandomized test of discrete data) cause the bar heights to increase, but their areas seem to be almost constant. Instead of using a histogram to assess uniformity, graph the empirical CDF. $\endgroup$
    – whuber ♦
    Commented Jul 21, 2015 at 23:45
  • 2
    $\begingroup$ Aside from the specific distribution, this question appears to be completely answered here $\endgroup$
    – Glen_b
    Commented Jul 22, 2015 at 9:43

1 Answer 1

Reset to default
10
$\begingroup$

The problem is the data are discrete so histograms can be deceiving. I coded a simulation with qqplots that show an approximate uniform distribution.

library(lattice)
set.seed(5545)
TotalNo=300
TotalYes=450

pvalueChi=rep(NA,10000)
pvalueFish=rep(NA,10000)

for(i in 1:10000){
  MaleAndNo=rbinom(1,TotalNo,.3)
  FemaleAndNo=TotalNo-MaleAndNo
  MaleAndYes=rbinom(1,TotalYes,.3)
  FemaleAndYes=TotalYes-MaleAndYes
  x=matrix(c(MaleAndNo,FemaleAndNo,MaleAndYes,FemaleAndYes),nrow=2,ncol=2)
  pvalueChi[i]=chisq.test(x)$p.value
  pvalueFish[i]=fisher.test(x)$p.value
}

dat=data.frame(pvalue=c(pvalueChi,pvalueFish),type=rep(c('Chi-Squared','Fishers'),each=10000))
histogram(~pvalue|type,data=dat,breaks=10)
qqmath(~pvalue|type,data=dat,distribution=qunif,
       panel = function(x, ...) {
         panel.qqmathline(x, ...)
         panel.qqmath(x, ...)
       })

enter image description here

enter image description here

Improve this answer
$\endgroup$
5
  • 5
    $\begingroup$ Such spikes and dips in histograms or bar charts of discrete data are often artifacts of the binning procedure. Don't trust them: use more discerning plots of the distributions, such as QQ plots or graphs of the ECDFs. Even if they are real, nobody will care provided the distributions of p-values are approximately uniform and of the right density where it matters for decision making: in the interval close to zero (and certainly less than 0.5). $\endgroup$
    – whuber ♦
    Commented Jul 21, 2015 at 23:47
  • $\begingroup$ Excellent point @whuber, I'll update with qqplots. $\endgroup$
    – Glen
    Commented Jul 22, 2015 at 0:02
  • 2
    $\begingroup$ @whuber, Glen, thanks so much! In fact the binning was deceptive, as simply splitting Glen's histograms into more breaks gave a similar pattern to mine. And I also get linear empirical CDF/QQ with my simulations, so the problem seems to be solved. $\endgroup$
    – juod
    Commented Jul 22, 2015 at 0:10
  • $\begingroup$ @juod: it would be highly appreciated if you could add the qqplot for illustration, maybe even for both simulations? $\endgroup$
    – Michael M
    Commented Jul 22, 2015 at 5:38
  • $\begingroup$ The qq plots really help--thank you. Don't you want to change the first paragraph of your answer, though? Do you still maintain there is a problem with the simulation and that there is a "spike" in the p-value distribution? $\endgroup$
    – whuber ♦
    Commented Jul 22, 2015 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.