Sampling distribution of the p-value

Question

In Bayesian Data Analysis 3 (page 151) in the sub-section P-values and u-values there is a part which reads:

In the special case that the parameters $\theta$ are known (or estimated to a very high precision) or in which the test statistic $T(y)$ is ancillary (that is, if it depends only on observed data and if its distribution is independent of the parameters of the model) with a continuous distribution, the posterior predictive p-value $\text{Pr}(T(y^{rep})>T(y) \mid y)$ has a distribution that is uniform if the model is true. Under these conditions, p-values less than 0.1 occur 10% of the time, p-values less than 0.05 occur 5% of the time, and so forth.

More generally, when posterior uncertainty in $\theta$ propagates to the distribution of $T(y|\theta)$, the distribution of the p-value, if the model is true, is more concentrated near the middle of the range: the p-value is more likely to be near 0.5 than near 0 or 1. (To be more precise, the sampling distribution of the p-value has been shown to be "stochastically less variable" than uniform.)

I don't understand the second paragraph and would greatly appreciate any further explanation on this point, or an example.

I think the first paragraph I've quoted refers to the "classical" p-value distribution under the null-hypothesis(?).

Gelman also has a paper on this topic which mentions the same point as above.

Answer 1

Suppose that we have a sample of $y$ and we are fitting the following model: $$y\sim N(\mu, 1).$$ From this we obtain the posterior of $\mu$. Now we generate $y^{rep}$ from the same distribution: $N(\mu, 1)$. What do we expect when comparing the density of $y$ and $y^{rep}$. If the model is correctly specified, the density of $y$ should be covered by the densities of $y^{rep}$.

In the figure below

we see that $y$ is not covered very well.

The question now is that why we need statistic $T$ in your quotation. Because, the graph just only gives us a qualitative idea. We need a number, that is ppp-value. In addition, comparing two distributions is not always easy as comparing to use statistics,

For example, suppose that $y$ is sampled from a skewed distribution but we fit $y$ as a $N(\mu, 1)$. Then skewness of $y$ is far from 0 while that of $y^{rep}$ is around 0 (since it is generated from $N(\mu, 1)$ (which is symmetrical)). In this case, of course, we fit a wrong model, and we can easily see that skewness of $y$ does not locate in the typical range of the skewness computed from $y^{rep}$. In other words, ppp-value is close to 0 or 1.