I am using the coin package in R to do a permutation test and it gives output that looks like this:
> pvalue(pt)
[1] 0.1479615
99 percent confidence interval:
0.1450811 0.1508754
Why would one want a confidence interval for a p-value? How should it be interpreted, and how should it be used?
The theory of permutation tests (and bootstraping) is based on the idea of looking at every possible permutation. If you can look at every possible permutation then you have an exact p-value.
For most cases it is impractical to compute every possible permutation so instead we sample from the "population" of possible permutations and compute a p-value from this "sample". So since we take a sample from the population we can use standard techniques (the binomial proportion in this case) to calculate a confidence interval for the true p-value that we would have calculated from every possible permutation.
How to use this confidence interval: If the value of alpha (your significance cutoff or probability of type I error) is in the interval then you are not sure whether to reject or not since a different sample of permutations could have given a different result. This means that you should probably rerun the analysis using a much larger number of permutations to get a better estimate of the true p-value.
If the entire interval is far from your alpha value then you can be confident in your rejection or nonrejection of the null hypothesis. Some people will report the bound of the confidence interval that is closest to alpha as a conservitive p-value. If you have done a large number of permutations then generally the bounds will be close enough together that it will not matter much either way.
The pure frequentist interpretation: If your were to do this experiment $N$ times, you should expect $X$ percent of the time (99% in your case) or $N * X$ times, that the true $p$ value falls inside the range provided by this 99% confidence interval that percentage or number of times.
