# Confidence interval for the p-value estimate when doing Monte Carlo testing

The second paragraph of the Monte Carlo testing section of the Wikipedia article on resampling statistics, the values of a confidence interval for a p-value from a MC sampling is given:

After $N$ random permutations, it is possible to obtain a confidence interval for the p-value based on the Binomial distribution. For example, if after $N = 10000$ random permutations the p-value is estimated to be $\hat{p}=0.05$, then a 99% confidence interval for the true $p$ (the one that would result from trying all possible permutations) is $[0.044, 0.056]$.

What's the formula to calculate it?

• Please flesh out your question so it can stand on its own without forcing readers to leave the site. – whuber Jul 13 '16 at 22:28

The article is talking about simulated p-values under resampling.

Let's specifically consider the case of resampling under a randomization test (it's somewhat easier to discuss).

Under the null, each resample has a probability of getting a test statistic at least as extreme as the one for the original sample that is equal to the permutation test p-value ($p$, say).

Consequently, out of $N$ resampling simulations, the number of resamples with a test statistic at least as extreme as the one in the original sample has a binomial distribution with parameters $N$ and $p$.

As a result, we can form a confidence interval for the p-value using standard methods for a binomial proportion confidence interval. The relevant Wikipedia page for the binomial proportion confidence interval discusses numerous methods (any of which would be suitable as long as $N$ is sufficiently large). Consider, for example, the usual interval based on a normal approximation.

If $X$ is the number of resamples with a test statistic at least as extreme as the one in the original sample then $\hat{p}=X/N$ will be approximately normal with mean $p$ and standard deviation $\sqrt{p(1-p)/N}$.

As a result an approximate 99% interval would be $\hat{p}\pm 2.576 \times \sqrt{\hat{p}(1-\hat{p})/N}$.

If $\hat{p}=0.05$ and $N=10000$ as in the example, that yields an interval of $(0.044,0.056)$ (to 2 significant figures) just as given in the article.

• +1. One caveat here is that if $X\approx 1$ or even $X=0$ (as I think often happens) then normal approximation is arguably not the best choice... – amoeba Jul 14 '16 at 12:42
• @amoeba Indeed not (hence the caveat in my answer); several of the other binomial intervals will usually be okay, but I didn't want to overcomplicate the answer with an extended discussion of binomial confidence intervals in the case of extreme $p$ --- and in any case it's usually of little consequence to give precise intervals then. – Glen_b Jul 14 '16 at 12:47
• +1. for the case that @amoeba mentioned, which other intervals do you recommend? Like Wilson score interval? Do you have a specific reference on this in mind that I can read in more depth? – NULL Apr 10 '19 at 17:02
• Another question, how would these estimates compare if we do bootstrap on the permutations? Is this even a thing? I mean, doing a lot of bootstrap on permutations, particularly in large scale inference settings, say to get a CI for p and FDR? – NULL Apr 10 '19 at 17:05