# Non-uniform distribution of p-values when simulating binomial tests under the null hypothesis

I heard that under the null hypothesis the p-value distribution should be uniform. However, simulations of binomial test in MATLAB return very different-from-uniform distributions with mean larger than 0.5 (0.518 in this case): coin = [0 1];
success_vec = nan(20000,1);

for i = 1:20000
success = 0;
for j = 1:200
success = success + coin(randperm(2,1));
end
success_vec(i) = success;
end

p_vec = binocdf(success_vec,200,0.5);
hist(p_vec);


Trying to change the way in which I generate random numbers didn't help. I would really appreciate any explanation here.

• One point to consider is that the p-values from the binomial test will take only certain discrete values (as the numerator is discrete): as an example, with only 20 trials [coin flips] per experiment, there are only 11 discrete p-values that can be returned. This is $n/2 + 1$ possible p-values, so with n = 200 trials per experiment, 101 discrete p-values. May 20, 2015 at 21:24
• Exactly what does Matlab's "binomial test" do?
– whuber
May 20, 2015 at 21:54
• It seems that this is the poster's binomial test, binocdf is just the CDF of the binomial uk.mathworks.com/help/stats/binocdf.html May 20, 2015 at 21:56

The result that $p$ values have a uniform distribution under $H_0$ holds for continuously distributed test statistics - at least for point nulls, as you have here.

As James Stanley mentions in comments the distribution of the test statistic is discrete, so that result doesn't apply. You may have no errors at all in your code (though I wouldn't display a discrete distribution with a histogram, I'd lean toward displaying the cdf or the pmf, or better, both).

While not actually uniform, each jump in the cdf of the p-value takes it to the line $F(x)=x$ (I don't know a name for this, but it ought to have a name, perhaps something like 'quasi-uniform'): It's quite possible to compute this distribution exactly, rather than simulate - but I've followed your lead and done a simulation (though a larger one than you have).

Such a distribution needn't have mean 0.5, though as the $n$ in the binomial increases the step cdf will approach the line more closely, and the mean will approach 0.5.

One implication of the discreteness of the p-values is that only certain significance levels are achievable -- the ones corresponding to the step-heights in the actual population cdf of p-values under the null. So for example you can have an $\alpha$ near 0.056 or one near 0.04, but not anything closer to 0.05.

• Thanks Glen and @JamesStanley! I'm trying to make sense of what exactly does it mean that the p-value distribution is not uniform, and what are the consequences in terms of hypothesis testing - but for that I guess I'll just dive into wikipedia :) May 21, 2015 at 10:43
• I'm not sure how well wikipedia covers that issue. One implication is that the discreteness of the set of p-values means only certain significance levels are achievable -- unless you use randomized tests (not randomization tests, which is something else again), but most people would prefer to have a limited set of $\alpha$s than do that. I'll edit a little into my answer. May 21, 2015 at 10:48
• @TanZor, you can look at it this way that there is an underlying uniform distribution on [0, 1], but because the binomial distribution is discrete, this uniform distribution has been non-uniformly discretized. Notice how the height of the bars in the pmf is proportional to the distance between neighboring bars – which is the same as noticing that the jumps take $F(x)$ to $x$. May 21, 2015 at 14:47
• A.Donda, Glen_b - thanks! You were a great help. May 21, 2015 at 21:47