Posterior predictive: what happens to integral over parameters?

Question

Question

I don't understand how when integrating over the parameters in the posterior predictive, the integration "disappears". It's hard for me to ask simply because I am confused, so here is an example.

Example

Imagine we have a Gaussian model with unknown mean $\mu$ and fixed variance $\sigma^2$. If $D$ is the training data and $D'$ is unseen data, then the posterior predictive is

$$ \begin{align} p(D' \mid D) &= \int p(D' \mid D, \mu) p(\mu \mid D) d\mu \\ &\stackrel{\star}{=} \int p(D' \mid \mu) p(\mu \mid D) d\mu \\ &\triangleq \int \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) d \mu \end{align} $$

where step $\star$ holds because the modeling assumption is that $D'$ is conditionally independent from $D$ given $\mu$ and the definitions of $\mu_N$ and $\sigma_N^2$ fall out of computing the posterior. See Bishop (2006) page 98 for details.

Here is where I am confused. I can show that

$$ \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) $$

Murphy's derivation in Conjugate Bayesian analysis of the Gaussian distribution suggests looking at Bishop, equation 2.115 (see my edit for more). My trouble is, Murphy then claims that

$$ p(D \mid D') = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) $$

which is what I mean by the integration should "disappearing". What happened? I understand that this new distribution has no dependence on $\mu$, but I would have expected

$$ \begin{align} p(D' \mid D) &= \int \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) d \mu \\ &= \int \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) d \mu \\ &= \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) \int d \mu \end{align} $$

But it's unclear what becomes of the integral. It's not a probability, so it's not like this is guaranteed to be 1.

---

Edit

This is my derivation based on Murphy's hint to look at Bishop (2006), page 93. Since both our posterior and prior are Gaussians, we can use the following fact:

$$ \begin{align} p(\textbf{x}) &= \mathcal{N}(\textbf{x} \mid \boldsymbol{\mu}, \boldsymbol{\Psi}) \\ p(\textbf{y} \mid \textbf{x}) &= \mathcal{N}(\textbf{y} \mid A \textbf{x} + \textbf{b}, \textbf{P}) \\ &\Downarrow \\ p(\textbf{y}) &= \mathcal{N}(\textbf{y} \mid \textbf{A} \boldsymbol{\mu} + \textbf{b}, \textbf{P} + \textbf{A} \boldsymbol{\Psi} \textbf{A}^{\top}) \end{align} $$

where we have

$$ \begin{align} \textbf{x} &= \mu \\ \boldsymbol{\mu} &= \mu_N \\ \boldsymbol{\Psi} &= \sigma_N^2 \\ \textbf{y} &= D' \\ \textbf{A} &= 1 \\ \textbf{b} &= 0 \\ \textbf{P} &= \sigma^2 \end{align} $$

This gives us

$$ \begin{align} p(\mu) &= \mathcal{N}(\mu \mid \mu_N, \mu_N^2) \\ p(D' \mid \mu) &= \mathcal{N}(D' \mid \mu, \sigma^2) \\ p(D' \mid \mu) p(\mu) = p(D') &= \mathcal{N}(D' \mid \mu_N, \sigma^2 + \sigma_N^2) \end{align} $$

We can add conditioning on $D$ at every step if we'd like, since it doesn't effect the distributions provided we have the parameters (i.i.d.):

$$ \begin{align} p(\mu \mid D) &= \mathcal{N}(\mu \mid \mu_N, \mu_N^2) \\ p(D' \mid \mu, D) &= \mathcal{N}(D' \mid \mu, \sigma^2) \\ p(D' \mid \mu, D) p(\mu) = p(D' \mid D) &= \mathcal{N}(D' \mid \mu_N, \sigma^2 + \sigma_N^2) \end{align} $$

Answer 1

If I understand correctly the question, it seems to me (and others before me in the comment section) that the derivation $$\mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2)$$ or equivalently $$p(D' \mid \mu, D) p(\mu) = p(D' \mid D) = \mathcal{N}(D' \mid \mu_N, \sigma^2 + \sigma_N^2)$$ is incorrect since the left hand side is a joint density on $(D',\mu)$ and the right hand side is a marginal density on $D'$. (When removing $\mu$ from the above rhs, it is as if $\mu$ is already integrated, but I advise against such loose reasonging.)

What is correct is that, if $$\underbrace{D'|\mu\sim}_\text{conditional}\mathcal{N}(\mu,\sigma^2)\qquad\text{and}\qquad\underbrace{\mu\sim}_\text{marginal}\mathcal{N}(\mu_N, \sigma_N^2)$$then $$\underbrace{D'\sim}_\text{marginal}\mathcal{N}(\mu_N, \sigma^2+\sigma_N^2)$$ as stated in the book.