The $p$-value is used to report how strongly we can presume against an hypothesis. As is clear, this $p$ value is itself estimated from data and if new data where collected in the same conditions, the new $p$ value will very unlikely be the same.
Halsey, Curran-Everett, Vowler & Drummond (2015) in a commentary to Nature Methods showed that the uncertainty surrounding a $p$-value can be fairly large. In a reply, Lazzeroni, Lu & Belitskaya-Lévy (2016, same journal) gave an example of an observed $p$ value of 0.049 whose confidence interval goes from 0.00000008 to 0.99.
My question is: do we know the sampling distribution of $p$ values? According to the latter, it does not depend on sample size (and presumably on the sample's standard deviation as all these are used to "standardize" the test statistic). Presumably, it might depend on the test procedure?
I know that if $H_0$ is true, the distribution of $p$-values is uniform over the range 0 to 1 (but can't remember where I learned this). As $H_0$ is more and more inadequate, the distribution of $p$-values becomes peaked, leaning over the 0% probabilities (for left-tail tests).
It is fairly easy with bootstrap to get a visual representation of the distribution of the $p$-values. However, a more satisfying answer would be to have a formula (closed-form is even better) so that we can know exactly what characteristics affect that distribution, and henceforth, the width of the confidence interval.
Do you know of such a formula, or if it is even possible to have one?