# Sampling distribution of the p-value

In Bayesian Data Analysis 3 (page 151) in the sub-section P-values and u-values there is a part which reads:

In the special case that the parameters $$\theta$$ are known (or estimated to a very high precision) or in which the test statistic $$T(y)$$ is ancillary (that is, if it depends only on observed data and if its distribution is independent of the parameters of the model) with a continuous distribution, the posterior predictive p-value $$\text{Pr}(T(y^{rep})>T(y) \mid y)$$ has a distribution that is uniform if the model is true. Under these conditions, p-values less than 0.1 occur 10% of the time, p-values less than 0.05 occur 5% of the time, and so forth.

More generally, when posterior uncertainty in $$\theta$$ propagates to the distribution of $$T(y|\theta)$$, the distribution of the p-value, if the model is true, is more concentrated near the middle of the range: the p-value is more likely to be near 0.5 than near 0 or 1. (To be more precise, the sampling distribution of the p-value has been shown to be "stochastically less variable" than uniform.)

I don't understand the second paragraph and would greatly appreciate any further explanation on this point, or an example.

I think the first paragraph I've quoted refers to the "classical" p-value distribution under the null-hypothesis(?).

Gelman also has a paper on this topic which mentions the same point as above.

• He says that people think that p-value distribution is uniform under the null hypothesis, but it is actually not uniform unless you know true value Jan 16, 2021 at 22:59

Suppose that we have a sample of $$y$$ and we are fitting the following model: $$y\sim N(\mu, 1).$$ From this we obtain the posterior of $$\mu$$. Now we generate $$y^{rep}$$ from the same distribution: $$N(\mu, 1)$$. What do we expect when comparing the density of $$y$$ and $$y^{rep}$$. If the model is correctly specified, the density of $$y$$ should be covered by the densities of $$y^{rep}$$.
we see that $$y$$ is not covered very well.
The question now is that why we need statistic $$T$$ in your quotation. Because, the graph just only gives us a qualitative idea. We need a number, that is ppp-value. In addition, comparing two distributions is not always easy as comparing to use statistics,
For example, suppose that $$y$$ is sampled from a skewed distribution but we fit $$y$$ as a $$N(\mu, 1)$$. Then skewness of $$y$$ is far from 0 while that of $$y^{rep}$$ is around 0 (since it is generated from $$N(\mu, 1)$$ (which is symmetrical)). In this case, of course, we fit a wrong model, and we can easily see that skewness of $$y$$ does not locate in the typical range of the skewness computed from $$y^{rep}$$. In other words, ppp-value is close to 0 or 1.