I came across this term in the deep learning book:

$p(x_{m+1}|x_1 ... x_m) = \int p(x_{m+1}|\theta)p(\theta|x_1 ... x_m)d\theta$

After some research I find that this term is the definition of the posterior predictive distribution. Is there a mathematical or intuitive proof of this? I'd prefer mathematical but anything helps right now.

EDIT: By proof, I am asking how LHS is equal to the RHS. For example, does this also work? If so, how do I arrive at PPD from the Bayes theorem?

$\frac{p(x_{m+1},x_1 ... x_m)}{p(x_1 ... x_m)} = \int p(x_{m+1}|\theta)p(\theta|x_1 ... x_m)d\theta$


1 Answer 1


You can't arrive directly from Bayes Theorem because the model has further assumptions. That is, given the parameter set $\theta$, $x_{m+1}$ is independent of previous data; However, if not given we can infer from previous samples which is what PPD is trying to do actually. Without any assumptions, the integral should have been the following: $$p(x_{m+1}|x_1,...,x_m)=\int p(x_{m+1}|\theta,x_1,...,x_m)p(\theta|x_1,...,x_m)d\theta$$ But, we summarize $p(x_{m+1}|\theta,x_1,...,x_m)$ as $p(x_{m+1}|\theta)$ assuming conditional independence. Your link has also the following sentence:

Given the assumption that the observed and unobserved data are conditional independent given $\theta$.

  • $\begingroup$ Thanks for the catch! I didn't notice that statement in the link. $\endgroup$ Sep 12, 2019 at 11:45
  • $\begingroup$ Followup question though: So, $x_{m+1}$ is not indenpendent of $x_1 ... x_m$; it's independent of $x_1 ... x_m$ given $\theta$. Is this correct? Because the former means that the LHS is just $p(x_{m+1})$. $\endgroup$ Sep 12, 2019 at 11:46
  • $\begingroup$ Yes, that's the whole idea. There is conditional independence, not full independence. $\endgroup$
    – gunes
    Sep 12, 2019 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.