I heard that under the null hypothesis the p-value distribution should be uniform. However, simulations of binomial test in MATLAB return very different-from-uniform distributions with mean larger than 0.5 (0.518 in this case): enter image description here

coin = [0 1];
success_vec = nan(20000,1);

for i = 1:20000
    success = 0;
    for j = 1:200
        success = success + coin(randperm(2,1));
    success_vec(i) = success;

    p_vec = binocdf(success_vec,200,0.5);

Trying to change the way in which I generate random numbers didn't help. I would really appreciate any explanation here.

  • 6
    $\begingroup$ One point to consider is that the p-values from the binomial test will take only certain discrete values (as the numerator is discrete): as an example, with only 20 trials [coin flips] per experiment, there are only 11 discrete p-values that can be returned. This is $n/2 + 1$ possible p-values, so with n = 200 trials per experiment, 101 discrete p-values. $\endgroup$ – James Stanley May 20 '15 at 21:24
  • $\begingroup$ Exactly what does Matlab's "binomial test" do? $\endgroup$ – whuber May 20 '15 at 21:54
  • 2
    $\begingroup$ It seems that this is the poster's binomial test, binocdf is just the CDF of the binomial uk.mathworks.com/help/stats/binocdf.html $\endgroup$ – conjugateprior May 20 '15 at 21:56

The result that $p$ values have a uniform distribution under $H_0$ holds for continuously distributed test statistics - at least for point nulls, as you have here.

As James Stanley mentions in comments the distribution of the test statistic is discrete, so that result doesn't apply. You may have no errors at all in your code (though I wouldn't display a discrete distribution with a histogram, I'd lean toward displaying the cdf or the pmf, or better, both).

While not actually uniform, each jump in the cdf of the p-value takes it to the line $F(x)=x$ (I don't know a name for this, but it ought to have a name, perhaps something like 'quasi-uniform'):

enter image description here

It's quite possible to compute this distribution exactly, rather than simulate - but I've followed your lead and done a simulation (though a larger one than you have).

Such a distribution needn't have mean 0.5, though as the $n$ in the binomial increases the step cdf will approach the line more closely, and the mean will approach 0.5.

One implication of the discreteness of the p-values is that only certain significance levels are achievable -- the ones corresponding to the step-heights in the actual population cdf of p-values under the null. So for example you can have an $\alpha$ near 0.056 or one near 0.04, but not anything closer to 0.05.

| cite | improve this answer | |
  • $\begingroup$ Thanks Glen and @JamesStanley! I'm trying to make sense of what exactly does it mean that the p-value distribution is not uniform, and what are the consequences in terms of hypothesis testing - but for that I guess I'll just dive into wikipedia :) $\endgroup$ – TanZor May 21 '15 at 10:43
  • 3
    $\begingroup$ I'm not sure how well wikipedia covers that issue. One implication is that the discreteness of the set of p-values means only certain significance levels are achievable -- unless you use randomized tests (not randomization tests, which is something else again), but most people would prefer to have a limited set of $\alpha$s than do that. I'll edit a little into my answer. $\endgroup$ – Glen_b May 21 '15 at 10:48
  • $\begingroup$ @TanZor, you can look at it this way that there is an underlying uniform distribution on [0, 1], but because the binomial distribution is discrete, this uniform distribution has been non-uniformly discretized. Notice how the height of the bars in the pmf is proportional to the distance between neighboring bars – which is the same as noticing that the jumps take $F(x)$ to $x$. $\endgroup$ – A. Donda May 21 '15 at 14:47
  • $\begingroup$ A.Donda, Glen_b - thanks! You were a great help. $\endgroup$ – TanZor May 21 '15 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.