The second paragraph of the Monte Carlo testing section of the Wikipedia article on resampling statistics, the values of a confidence interval for a p-value from a MC sampling is given:

After $N$ random permutations, it is possible to obtain a confidence interval for the p-value based on the Binomial distribution. For example, if after $N = 10000$ random permutations the p-value is estimated to be $\hat{p}=0.05$, then a 99% confidence interval for the true $p$ (the one that would result from trying all possible permutations) is $[0.044, 0.056]$.

What's the formula to calculate it?

  • 1
    $\begingroup$ Please flesh out your question so it can stand on its own without forcing readers to leave the site. $\endgroup$ – whuber Jul 13 '16 at 22:28

The article is talking about simulated p-values under resampling.

Let's specifically consider the case of resampling under a randomization test (it's somewhat easier to discuss).

Under the null, each resample has a probability of getting a test statistic at least as extreme as the one for the original sample that is equal to the permutation test p-value ($p$, say).

Consequently, out of $N$ resampling simulations, the number of resamples with a test statistic at least as extreme as the one in the original sample has a binomial distribution with parameters $N$ and $p$.

As a result, we can form a confidence interval for the p-value using standard methods for a binomial proportion confidence interval. The relevant Wikipedia page for the binomial proportion confidence interval discusses numerous methods (any of which would be suitable as long as $N$ is sufficiently large). Consider, for example, the usual interval based on a normal approximation.

If $X$ is the number of resamples with a test statistic at least as extreme as the one in the original sample then $\hat{p}=X/N$ will be approximately normal with mean $p$ and standard deviation $\sqrt{p(1-p)/N}$.

As a result an approximate 99% interval would be $\hat{p}\pm 2.576 \times \sqrt{\hat{p}(1-\hat{p})/N}$.

If $\hat{p}=0.05$ and $N=10000$ as in the example, that yields an interval of $(0.044,0.056)$ (to 2 significant figures) just as given in the article.

  • 2
    $\begingroup$ +1. One caveat here is that if $X\approx 1$ or even $X=0$ (as I think often happens) then normal approximation is arguably not the best choice... $\endgroup$ – amoeba Jul 14 '16 at 12:42
  • 3
    $\begingroup$ @amoeba Indeed not (hence the caveat in my answer); several of the other binomial intervals will usually be okay, but I didn't want to overcomplicate the answer with an extended discussion of binomial confidence intervals in the case of extreme $p$ --- and in any case it's usually of little consequence to give precise intervals then. $\endgroup$ – Glen_b Jul 14 '16 at 12:47
  • $\begingroup$ +1. for the case that @amoeba mentioned, which other intervals do you recommend? Like Wilson score interval? Do you have a specific reference on this in mind that I can read in more depth? $\endgroup$ – NULL Apr 10 '19 at 17:02
  • $\begingroup$ Another question, how would these estimates compare if we do bootstrap on the permutations? Is this even a thing? I mean, doing a lot of bootstrap on permutations, particularly in large scale inference settings, say to get a CI for p and FDR? $\endgroup$ – NULL Apr 10 '19 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.