In Bayesian Data Analysis 3 (page 151) in the sub-section P-values and u-values there is a part which reads:

In the special case that the parameters $\theta$ are known (or estimated to a very high precision) or in which the test statistic $T(y)$ is ancillary (that is, if it depends only on observed data and if its distribution is independent of the parameters of the model) with a continuous distribution, the posterior predictive p-value $\text{Pr}(T(y^{rep})>T(y) \mid y)$ has a distribution that is uniform if the model is true. Under these conditions, p-values less than 0.1 occur 10% of the time, p-values less than 0.05 occur 5% of the time, and so forth.

More generally, when posterior uncertainty in $\theta$ propagates to the distribution of $T(y|\theta)$, the distribution of the p-value, if the model is true, is more concentrated near the middle of the range: the p-value is more likely to be near 0.5 than near 0 or 1. (To be more precise, the sampling distribution of the p-value has been shown to be "stochastically less variable" than uniform.)

I don't understand the second paragraph and would greatly appreciate any further explanation on this point, or an example.

I think the first paragraph I've quoted refers to the "classical" p-value distribution under the null-hypothesis(?).

Gelman also has a paper on this topic which mentions the same point as above.

  • $\begingroup$ He says that people think that p-value distribution is uniform under the null hypothesis, but it is actually not uniform unless you know true value $\endgroup$
    – Aksakal
    Jan 16, 2021 at 22:59

1 Answer 1


Suppose that we have a sample of $y$ and we are fitting the following model: $$y\sim N(\mu, 1).$$ From this we obtain the posterior of $\mu$. Now we generate $y^{rep}$ from the same distribution: $N(\mu, 1)$. What do we expect when comparing the density of $y$ and $y^{rep}$. If the model is correctly specified, the density of $y$ should be covered by the densities of $y^{rep}$.

In the figure below

enter image description here

we see that $y$ is not covered very well.

The question now is that why we need statistic $T$ in your quotation. Because, the graph just only gives us a qualitative idea. We need a number, that is ppp-value. In addition, comparing two distributions is not always easy as comparing to use statistics,

For example, suppose that $y$ is sampled from a skewed distribution but we fit $y$ as a $N(\mu, 1)$. Then skewness of $y$ is far from 0 while that of $y^{rep}$ is around 0 (since it is generated from $N(\mu, 1)$ (which is symmetrical)). In this case, of course, we fit a wrong model, and we can easily see that skewness of $y$ does not locate in the typical range of the skewness computed from $y^{rep}$. In other words, ppp-value is close to 0 or 1.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.