I am wondering if anyone can check my understanding of the following passage concerning posterior predictive p-values in the textbook "Bayesian Data Analysis 3rd Edition" on page 151:

In the special case that the parameters $\theta$ are known (or estimated to a very high precision) or in which the test statistic $T(y)$ is ancillary (that is, if it depends only on observed data and if its distribution is independent of the parameters of the model) with a continuous distribution, the posterior predictive p-value $P(T(y^{\text{rep}}) > T(y) \mid y)$ has a distribution that is uniform if the model is true.

I suspect this is something akin to what is called the "CDF transformation" where a random variable $X$ is transformed by plugging it into its own CDF. In other words, if $X$ has a cdf $F_X( \cdot)$, then the random variable $F_X(X)$ is uniformly distributed. And correct me if this is wrong, but another way to write this is $P(X \le x \mid \tilde{X} = x)$ where $\tilde{X}$ is just another iid copy of $X$.

So I guess if we can show that

  • $T(y^{\text{rep}}) \mid y$ and
  • $T(y) \mid \theta$

are distributed in the same way, that this implies

$$ P(T(y^{\text{rep}}) > T(y) \mid y) = 1 - F(T(y)) $$

where $F$ is the CDF of both of those random variables. Is that right?

  • 1
    $\begingroup$ Yes, this is correct, as a function of the rv $Y$, in this special case, $\mathbb{P}(T(Y^{\text{rep}}) > T(Y) \mid Y)$ is distributed as a Uniform. $\endgroup$ – Xi'an Jan 10 '19 at 7:50

Thanks to @Xi'an for the confirmation. The authors are saying that when $$ P(T(y^{\text{rep}}) > T(y) \mid y) = P(T(y^{\text{rep}}) > T(y) \mid \theta, y) $$ then the reasoning of the CDF transformation mentioned above implies that these are uniformly distributed on the unit interval.

I was planning on asking about the next paragraph as well, but I think I understand it now.

More generally, when posterior uncertainty in $\theta$ propagates to the distribution of $T(y \mid \theta)$, the distribution of the p-value is more likely to be near $.5$ than near $0$ or $1$. (To be more precise, the sampling distribution of the p-value has been shown to be 'stochastically less variable' than uniform.)

Perhaps by "stochastically less variable" they mean convex-ordered. If this is the case, it's straightforward to show, using Jensen's inequality, that conditional probabilities are convex-ordered when one conditions on less than the other.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.