# What is the confidence interval of a p-value?

The $p$-value is used to report how strongly we can presume against an hypothesis. As is clear, this $p$ value is itself estimated from data and if new data where collected in the same conditions, the new $p$ value will very unlikely be the same.

Halsey, Curran-Everett, Vowler & Drummond (2015) in a commentary to Nature Methods showed that the uncertainty surrounding a $p$-value can be fairly large. In a reply, Lazzeroni, Lu & Belitskaya-Lévy (2016, same journal) gave an example of an observed $p$ value of 0.049 whose confidence interval goes from 0.00000008 to 0.99.

My question is: do we know the sampling distribution of $p$ values? According to the latter, it does not depend on sample size (and presumably on the sample's standard deviation as all these are used to "standardize" the test statistic). Presumably, it might depend on the test procedure?

I know that if $H_0$ is true, the distribution of $p$-values is uniform over the range 0 to 1 (but can't remember where I learned this). As $H_0$ is more and more inadequate, the distribution of $p$-values becomes peaked, leaning over the 0% probabilities (for left-tail tests).

It is fairly easy with bootstrap to get a visual representation of the distribution of the $p$-values. However, a more satisfying answer would be to have a formula (closed-form is even better) so that we can know exactly what characteristics affect that distribution, and henceforth, the width of the confidence interval.

Do you know of such a formula, or if it is even possible to have one?

• I believe what you want would be a prediction interval for future p-values constructed under the same conditions as the original p-value? Perhaps you do mean confidence interval rather than prediction interval, but talking about a confidence interval for an observed value is very confusing to me. Whether you meant prediction or confidence interval, I'm pretty sure you want to specify that the interval refers to the mean of future p-values from future studies. – Cliff AB Jan 4 '17 at 20:17
• @Cliff If you accept that there is a sampling distribution of p-values (which seems uncontroversial), then the fact that p-values are bounded implies this sampling distribution has an expectation. Its expectation evidently is a property of the underlying distribution within the context of a specific model and specific test statistic. Given that, it looks like this expectation could reasonably viewed as a property of the distribution itself, permitting one to apply all the conventional concepts of estimate, estimator, and confidence interval. – whuber Jan 4 '17 at 20:27
• Halsey et al paper that OP mentioned and the reasoning behind it is discussed at great length in this recent thread: stats.stackexchange.com/questions/250269 - which I would say is perhaps even a duplicate (@whuber). The general conclusion of that thread is Halsey et al (who borrow their claims from the earlier work by Cumming) are sloppy and do not state their assumptions. I strongly dislike their paper. – amoeba says Reinstate Monica Jan 4 '17 at 20:31
• @whuber Yes, I agree. Still it might be useful for the OP to read those discussions. – amoeba says Reinstate Monica Jan 4 '17 at 20:44
• @Amoeba Be careful: one does not construct a CI for a statistic; a CI refers to a parameter. In classical situations (Z tests, t tests, etc) there is a one-to-one correspondence between the statistic and the p-value. To the extent a statistic can estimate something (typically an effect size), a fortiori a p-value must be estimating something, too. But what it might estimating has nothing to do with how one constructs a CI. A plausible candidate for its estimand is the expected p-value (for a given model, given statistic, and given effect size). The chief difficulty, it seems to me, – whuber Jan 4 '17 at 20:58

• Hello @David, glad to see you on StackExchange. Although I agree with you that in general, p is not a parameter, I am sure we could image a world in which populations are characterized by a parameter $\pi$. In this world, all the samples would have a constant size and all the sampling method will be constant as well. In this improbable world (if you allow the pun), $\pi$ is a parameter, and $p \equiv \hat\pi$, probably the best, unbiased estimate of $\pi$. Hence, if I frame my question relative to this world, can we have a confidence interval around an observed $p$? – Denis Cousineau Jan 4 '17 at 20:12
• Sure, as @David says, $p$ are informative. They are just variable. If we could get some confidence interval, and find that the whole interval is very narrow and close to zero, that would add additional strength to a conclusion. – Denis Cousineau Jan 4 '17 at 20:32