I came across this term in the deep learning book:

$p(x_{m+1}|x_1 ... x_m) = \int p(x_{m+1}|\theta)p(\theta|x_1 ... x_m)d\theta$

After some research I find that this term is the definition of the posterior predictive distribution. Is there a mathematical or intuitive proof of this? I'd prefer mathematical but anything helps right now.

EDIT: By proof, I am asking how LHS is equal to the RHS. For example, does this also work? If so, how do I arrive at PPD from the Bayes theorem?

$\frac{p(x_{m+1},x_1 ... x_m)}{p(x_1 ... x_m)} = \int p(x_{m+1}|\theta)p(\theta|x_1 ... x_m)d\theta$


You can't arrive directly from Bayes Theorem because the model has further assumptions. That is, given the parameter set $\theta$, $x_{m+1}$ is independent of previous data; However, if not given we can infer from previous samples which is what PPD is trying to do actually. Without any assumptions, the integral should have been the following: $$p(x_{m+1}|x_1,...,x_m)=\int p(x_{m+1}|\theta,x_1,...,x_m)p(\theta|x_1,...,x_m)d\theta$$ But, we summarize $p(x_{m+1}|\theta,x_1,...,x_m)$ as $p(x_{m+1}|\theta)$ assuming conditional independence. Your link has also the following sentence:

Given the assumption that the observed and unobserved data are conditional independent given $\theta$.

| cite | improve this answer | |
  • $\begingroup$ Thanks for the catch! I didn't notice that statement in the link. $\endgroup$ – SPRajagopal Sep 12 '19 at 11:45
  • $\begingroup$ Followup question though: So, $x_{m+1}$ is not indenpendent of $x_1 ... x_m$; it's independent of $x_1 ... x_m$ given $\theta$. Is this correct? Because the former means that the LHS is just $p(x_{m+1})$. $\endgroup$ – SPRajagopal Sep 12 '19 at 11:46
  • $\begingroup$ Yes, that's the whole idea. There is conditional independence, not full independence. $\endgroup$ – gunes Sep 12 '19 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.