Predicted distribution over the new data sample

Question

I'm reading Deep Learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville. In chapter 5 about Bayesian Statistics, to find the distribution over the new data sample $x^{m+1}$ after observing $m$ samples $x^{1}, x^{2}, ..., x^{m}$ we write (equation 5.68 on page 134),

$p(x^{m+1}|x^{1}, x^{2}, ..., x^{m}) = \int p(x^{m+1}|\theta)p(\theta|x^{1}, x^{2}, ..., x^{m})d\theta$

How did they derive this ? The only thing I know which connects the data samples ($x$'s) and the model parameter ($\theta$) is the following

$p(\theta|x^{1}, x^{2}, ..., x^{m})=\frac{p(x^{1}, x^{2}, ..., x^{m}|\theta)p(\theta)}{p(x^{1}, x^{2}, ..., x^{m})}$

From the integral in the equation, I think something is being marginalised. But how exactly should this be derived ? I tried applying all the tools that I know (chain rule, Bayes rule), but I could not understand or derive this.

Given this is not the first question I'm posting on math and cross validated SE related to probability (not related to machine learning as such), I do not have a very good understanding of the subject. I do not have the right intuition for such equations involving probabilities and probability distributions. Could you also please explain what is the equation supposed to mean ?

Answer 1

"How did they derive this?"

This is called the posterior predictive distribution (ppd):

\begin{align*} &p(x^{m+1}|x^{1}, x^{2}, ..., x^{m}) \\ &= \int p(x^{m+1}, \theta|x^{1}, x^{2}, ..., x^{m}) d\theta \tag{defn marginal dstn.}\\ &= \int p(x^{m+1} | \theta, x^{1}, x^{2}, ..., x^{m}) p(\theta|x^{1}, x^{2}, ..., x^{m}) d\theta \tag{defn cndtl prob}\\ &= \int \underbrace{p(x^{m+1}|\theta)}_{(1)}\underbrace{p(\theta|x^{1}, x^{2}, ..., x^{m})}_{(2)}d\theta. \tag{cndtl indep} \end{align*}

The goal is to write what you want (the ppd) in terms of what you know ((1) and (2)). (1) is the likelihood, and (2) is the posterior. If your parameters aren't continuous rvs, then replace the integrals with sums.