How to interpret Bayesian (posterior predictive) p-value of 0.5?

In the following paper found here and reference below, the author suggests that "if the model is true or close to true, the posterior predictive p-value will almost certainly be very close to 0.5" . This is found in the very beginning of Section 6 in the paper.

I am trying to interpret what is meant when he says 'model is true'. My questions are:

i) Statistically what is a "true model" as said in the quote above?

ii) What does a value of 0.5 mean in simple words?

Gelman, A. (2013). Two simple examples for understanding posterior p-values whose distributions are far from uniform. Electronic Journal of Statistics, 7, 2595-2602.

• Could you give a full reference to the paper just in case the link goes dead in the future (so as to make the thread comprehensible for future readers)? Aug 17 '18 at 7:23

The model is true if the data are generated according to the model you are doing inference with. In other words, the unobserved parameter is generated by the prior, and then, using that parameter draw, your observed data are generated by the likelihood. This is not the setup where you consider multiple models $$M_1, M_2, \ldots$$ and have discrete probability distributions describing the model uncertainty.

A posterior predictive "p-value" of .5 means your test statistic $$T(y)$$ will be exactly equal to the median of the posterior predictive distribution of $$T(y^{\text{rep}})$$. Generally, this distribution and its median are obtained by looking at simulated data. Roughly speaking, this tells us that predictions (i.e. $$T(y^{\text{rep}})$$) "look like" our real data $$T(y)$$. If our model's predictions are "biased" to be too high, then we will get a number greater than $$.5$$, and if they are generally on the low side, we will get a number less than $$.5$$.

The posterior predictive distribution is \begin{align*} p(y^{\text{rep}} \mid y) &= \int p(y^{\text{rep}},\theta \mid y) d\theta\\ &= \int p(y^{\text{rep}}\mid \theta, y)p(\theta \mid y) d\theta\\ &= \int \underbrace{p(y^{\text{rep}} \mid \theta)}_{\text{model}}\underbrace{p(\theta \mid y)}_{\text{posterior}} d\theta \\ &= E_{\theta \mid y}\left[p(y^{\text{rep}} \mid \theta) \right]. \end{align*}

Then you take this distribution and integrate over the region where $$T(y^{\text{rep}})$$ is greater than some calculated nonrandom statistic of the dataset $$T(y)$$.

$$P(T(y^{\text{rep}}) > T(y) \mid y) = \int_{\{T(y^{\text{rep}}) : T(y^{\text{rep}}) > T(y) \}} p(y^{\text{rep}} \mid y) dy^{\text{rep}}.$$

In practice, if computing the above integral is too difficult, this means drawing parameters from the posterior, and then, using these parameters, simulating many $$y^{\text{rep}}$$s. For each simulated data set (of the same size as your original/real data set), you calculate $$T(y^{\text{rep}})$$. Then you calculate what percent of these simulated values are above your single $$T(y)$$ coming from your real data set.

Because you are assuming there is no model uncertainty, $$p(y^{\text{rep}} \mid y)$$ is an integral over the parameter space; not the parameter space AND the model space.

• if I do a ppd check Pr(T(y_rep)> T(y)), and get 0.5,I know this means the simulated and observed data are similar and the model is a good fit. But my question is why? what does the 0.5 tell us as say opposed to 0.8? Aug 17 '18 at 4:08
• @user121 it might to help to think about the situation when it comes out to be .0001. This is bad because your model is predicting $T$s that are nowhere near the $T$ you calculated from the data. Aug 17 '18 at 4:10
• what would .9999 imply? Aug 17 '18 at 4:12
• It’s far away (as measured by your posterior) in the other direction. It means your model is predicting something not like the data you have, and it’s impoetant to nderstand why it’s overriding common sense/intuition. .5 is good because half of the simulations are above your one real T and half are below. Aug 17 '18 at 4:16
• appreciate the feedback. I dont think I am asking clearly what I'm looking for. If the p-val= 0.9999, what statement can we make versus a p-val =0.5214. I do not understand why 0.5 is the 'ideal' number for model fit. It is something fundamental I'm missing. Is there some physical meaning behind it? Aug 17 '18 at 4:23

I would recommend reading the underlying papers that this paper is derived from as the terminology doesn't appear to have become standard in the field. The original paper is by Rubin, but Gelman is writing from Meng.

Meng, X. (1994). Posterior Predictive p-Values. The Annals of Statistics, 22(3), 1142-1160.

I am trying to interpret what is meant is meant when he says 'model is true'. My questions are:

i) Statistically, what is a "true model" as said in the quote above?

ii) What does a value of 0.5 mean in simple words?

So there is some unfortunate language usage as p-values are a Frequentist idea and Bayesian methods do not have p-values. Nonetheless, within the context of the articles beginning with Rubin, we can discuss the idea of a Bayesian p-value in a narrow sense.

As to question one, Bayesian models do not falsify a null hypothesis. In fact, except where some method is intending to mimic Frequentist methods, as in this paper, the phrasing "null hypothesis" is rarely used. Instead, Bayesian methods are generative methods and are usually constructed from a different set of axioms.

The easiest way to approach your question is from Cox's axioms.

Cox, R. T. (1961). The Algebra of Probable Inference. Baltimore, MD: Johns Hopkins University Press.

Cox's first axiom is that plausibilities of a proposition are a real number that varies with the information related to the proposition. Notice the word probability wasn't used as this also allows us to think in terms of odds or other mechanisms. This varies very strongly from null hypothesis methods. To see an example, consider binary hypotheses $$H_1,H_2$$, which in Frequentist methods will be denoted $$H_0,H_A$$. What is the difference?

$$H_0$$ is conditioned to be true and the p-value tests the probability of observing the sample, given the null is true. It does not test if the null is actually true or false and $$H_A$$ has no form of probability statement attached to it. So, if $$p<.05$$, this does not imply that $$\Pr(H_A)>.95$$.

In the Bayesian framework, each proposition has a probability assigned to it so that if $$H_1:\mu\le{k}$$ and $$H_2:\mu>k$$, then it follows that if $$\Pr(H_1)=.7327$$ then $$\Pr(H_2)=.2673$$.

The true model is the model that generated the data in nature. This varies from the Frequentist method which depends only on the sampling distribution of the statistic, generally.

As to question two, Gelman is responding to Meng. He was pointing out that in a broad variety of circumstances if the null hypothesis is true, then $$\Pr(y^{rep}|y)$$ will cluster around .5 if you average over the sample space. He provides a case where this is useful and one where it is a problem. However, the hint as to why comes from the examples, particularly the first.

The first has been constructed so that there are great prior uncertainty and this use of a nearly uninformative prior propagates through to the predictive distribution in such a way that, almost regardless of your sample, Rubin and Ming's posterior predictive p-values will be near 50%. In this case, it would mean that it would tell you there is a 50% chance the null is true, which is highly undesirable since you would rather be either near 100% or in the case of falsehood, 0%.

The idea of Bayesian posterior p-values is the observation that since you are now in the sample space as random, rather than the parameter space as random, the rough interpretation of a Bayesian posterior probability is remarkably close to the Frequentist p-value. It is problematic because the model is not considered a parameter, in itself, and has no prior probability as would be the case in a test of many different models. The model, $$H_A$$ versus $$H_B$$ is implicit.

This article is a warning of something that should, in a sense, be obvious. Imagine you had fifty million data points and there was no ambiguity as to the location of the parameter, then you would be stunned if the resulting predictive distribution was a bad estimator over the sample space. Now consider a model where the results are ambiguous and the prior was at best weakly informative, then even if the model is true, it would be surprising to get a clear result from the posterior predictive distribution.

He provides an example where data is drawn from a population that has a standard normal distribution. The required sample would have to be 28,000 to get a rejection of the model. In a standard normal population, that will never happen.

The model is about the propagation of uncertainty and whether or not Rubin/Meng's idea generates a useful construct when it is needed most when the data is poor, small, weak or ambiguous as opposed to samples that are stunningly clear. As an out-of-sample test tool, its sampling properties are undesirable in some circumstances, but desirable in others.

In this case, what Gelman is saying is that regardless of the true probability of the model, the out-of-sample validation score provided by the Bayesian posterior predictive p-value will be near 50% when the null is true when the data doesn't clearly point to a solution.

This has lead to the criticism of the idea as uncalibrated with the true probabilities. See

Bayarri, M. J. and Berger, J. (2000). P-values for composite null models. Journal of the American Statistical Association 95, 1127–1142.

• very good feedback Aug 17 '18 at 4:49

On question (ii), there is much more to be said. Indeed, there are other kinds of "Bayesian" GOF p-values that have uniform distribution when the model used for inference was used to generate the data -which means that such p-values should not be compared to 0.5 but to 0 and 1. The simplest of these is the sampled posterio GOF p-value. See the following refernces:

Robins JM, van der Vaart A, Ventura V (2000) Asymptotic distribution of P values in composite null models. J Am Stat Assoc 95: 1143–56.

Johnson VE (2007) Bayesian Model Assessment Using Pivotal Quantities. Bayesian Anal 2: 719–34.

Gosselin F. (2011) A New Calibrated Bayesian Internal Goodness-of-Fit Method: Sampled Posterior p-Values as Simple and General p-Values That Allow Double Use of the Data. Plos One. https://doi.org/10.1371/journal.pone.0014770