I'm reading Deep Learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville. In chapter 5 about Bayesian Statistics, to find the distribution over the new data sample $x^{m+1}$ after observing $m$ samples $x^{1}, x^{2}, ..., x^{m}$ we write (equation 5.68 on page 134),

$p(x^{m+1}|x^{1}, x^{2}, ..., x^{m}) = \int p(x^{m+1}|\theta)p(\theta|x^{1}, x^{2}, ..., x^{m})d\theta$

How did they derive this ? The only thing I know which connects the data samples ($x$'s) and the model parameter ($\theta$) is the following

$p(\theta|x^{1}, x^{2}, ..., x^{m})=\frac{p(x^{1}, x^{2}, ..., x^{m}|\theta)p(\theta)}{p(x^{1}, x^{2}, ..., x^{m})}$

From the integral in the equation, I think something is being marginalised. But how exactly should this be derived ? I tried applying all the tools that I know (chain rule, Bayes rule), but I could not understand or derive this.

Given this is not the first question I'm posting on math and cross validated SE related to probability (not related to machine learning as such), I do not have a very good understanding of the subject. I do not have the right intuition for such equations involving probabilities and probability distributions. Could you also please explain what is the equation supposed to mean ?


1 Answer 1


"How did they derive this?"

This is called the posterior predictive distribution (ppd):

\begin{align*} &p(x^{m+1}|x^{1}, x^{2}, ..., x^{m}) \\ &= \int p(x^{m+1}, \theta|x^{1}, x^{2}, ..., x^{m}) d\theta \tag{defn marginal dstn.}\\ &= \int p(x^{m+1} | \theta, x^{1}, x^{2}, ..., x^{m}) p(\theta|x^{1}, x^{2}, ..., x^{m}) d\theta \tag{defn cndtl prob}\\ &= \int \underbrace{p(x^{m+1}|\theta)}_{(1)}\underbrace{p(\theta|x^{1}, x^{2}, ..., x^{m})}_{(2)}d\theta. \tag{cndtl indep} \end{align*}

The goal is to write what you want (the ppd) in terms of what you know ((1) and (2)). (1) is the likelihood, and (2) is the posterior. If your parameters aren't continuous rvs, then replace the integrals with sums.

  • $\begingroup$ Thanks for taking the time to answer. Is the first step supposed to mean: $\int p((x^{m+1}, \theta)|x^{1}, x^{2}, ..., x^{m})d\theta$ (I'm just adding a bracket for clarity) that is, the joint distribution of $x^{m+1}$ and $\theta$ given $x^{1}, x^{2}, ..., x^{m}$ ? Could you please elaborate the second step ? This is the only definition of conditional probability I know: $P(A|B) = \frac{P(A,B)}{P(B)}$. I don't see: $\frac{p(x^{m+1}, \theta, x^{1}, x^{2}, ..., x^{m})}{x^{1}, x^{2}, ..., x^{m}}$, clearly I misunderstood something. Could you please explain ? I understood the last step! $\endgroup$
    – rranjik
    Apr 6, 2019 at 18:53
  • $\begingroup$ @rranjik (1) yep that’s right. (2) use that definition twice and you’ll see a term cancel out. $\endgroup$
    – Taylor
    Apr 6, 2019 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.