Timeline for Fisher's exact test gives non-uniform p-values
Current License: CC BY-SA 3.0
19 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 22, 2015 at 9:43 | comment | added | Glen_b | Aside from the specific distribution, this question appears to be completely answered here | |
| Jul 22, 2015 at 9:38 | history | edited | juod | CC BY-SA 3.0 |
added 451 characters in body
|
| Jul 22, 2015 at 2:08 | history | tweeted | twitter.com/#!/StackStats/status/623675987994247168 | ||
| Jul 22, 2015 at 0:12 | vote | accept | juod | ||
| Jul 21, 2015 at 23:45 | comment | added | whuber♦ | These histograms look remarkably uniform to me. You have to remember that histograms display probability (or frequency) by means of area. The increasing gaps to the right (due to the unavoidable discreteness of the p-value distribution of any nonrandomized test of discrete data) cause the bar heights to increase, but their areas seem to be almost constant. Instead of using a histogram to assess uniformity, graph the empirical CDF. | |
| Jul 21, 2015 at 23:35 | answer | added | Glen | timeline score: 10 | |
| Jul 21, 2015 at 23:22 | comment | added | bdeonovic | @juod I see, then Glen's comment stands to explain the phenomenon: p-values are uniformly distributed under the null hypothesis, which does not hold in each of your simulation iterations. | |
| Jul 21, 2015 at 23:08 | comment | added | juod | @Glen, bdeonovic: Let's say sample 1 has some number A of 1s, and sample 2 has some number B of 1s. A and B are not equal here - if it were so, every time the test statistic would be the same - but A+B stays constant. In other words, it's always 500 zeros and 500 ones, but sometimes they are divided equally between the samples, and sometimes they are not. Thinking this way, "no difference between samples" is probably the most frequent case - but it should be reflected in the test statistic, not the p-value... Am I understanding this correctly? | |
| Jul 21, 2015 at 22:55 | comment | added | bdeonovic | @Glen it seems to me from the code that in each iteration both samples have the same number of 0s and 1s (i.e. null hypothesis should hold) or am I wrong? | |
| Jul 21, 2015 at 22:34 | comment | added | Glen | The p-value is uniformly distributed if the null hypothesis is true. That's probably not the case for a majority of the tests. | |
| Jul 21, 2015 at 22:10 | comment | added | gung - Reinstate Monica |
You should set the seed for reproducibility. I used set.seed(6069). I get a histogram with the same profile as yours, but I notice that mean(p<.05) # [1] 0.0516. I'll do more later.
|
|
| Jul 21, 2015 at 21:48 | comment | added | juod | Added the R code. I get similar results if I export the columns of randomized 0/1s and calculate Fisher's test in a genetic software called PLINK - I was thinking, maybe it's some property of the test itself?.. | |
| Jul 21, 2015 at 21:44 | history | edited | juod | CC BY-SA 3.0 |
added 299 characters in body
|
| Jul 21, 2015 at 20:47 | comment | added | conjugateprior | It might be useful to have look at your R code. | |
| Jul 21, 2015 at 20:37 | history | edited | juod | CC BY-SA 3.0 |
added 300 characters in body
|
| Jul 21, 2015 at 20:30 | comment | added | juod | Thanks for the suggestion, although it didn't solve the problem. I'll update the question right away | |
| Jul 21, 2015 at 19:57 | comment | added | Michael M | Try to repeat your simulation while holding not only the group sizes (500 each) but also the sum of "1" (over the pooled sample) constant. The p value of Fisher's exact test is derived under this "fixed marginal distribution" setting. Does the picture look better then? Btw. you cannot expect the p-value distribution to be exactly uniform by the discrete nature of the sampling distribution (i.e. the hypergeometric). | |
| Jul 21, 2015 at 19:43 | review | First posts | |||
| Jul 21, 2015 at 19:53 | |||||
| Jul 21, 2015 at 19:43 | history | asked | juod | CC BY-SA 3.0 |