# Posterior predictive: what happens to integral over parameters?

### Question

I don't understand how when integrating over the parameters in the posterior predictive, the integration "disappears". It's hard for me to ask simply because I am confused, so here is an example.

### Example

Imagine we have a Gaussian model with unknown mean $$\mu$$ and fixed variance $$\sigma^2$$. If $$D$$ is the training data and $$D'$$ is unseen data, then the posterior predictive is

\begin{align} p(D' \mid D) &= \int p(D' \mid D, \mu) p(\mu \mid D) d\mu \\ &\stackrel{\star}{=} \int p(D' \mid \mu) p(\mu \mid D) d\mu \\ &\triangleq \int \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) d \mu \end{align}

where step $$\star$$ holds because the modeling assumption is that $$D'$$ is conditionally independent from $$D$$ given $$\mu$$ and the definitions of $$\mu_N$$ and $$\sigma_N^2$$ fall out of computing the posterior. See Bishop (2006) page 98 for details.

Here is where I am confused. I can show that

$$\mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2)$$

Murphy's derivation in Conjugate Bayesian analysis of the Gaussian distribution suggests looking at Bishop, equation 2.115 (see my edit for more). My trouble is, Murphy then claims that

$$p(D \mid D') = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2)$$

which is what I mean by the integration should "disappearing". What happened? I understand that this new distribution has no dependence on $$\mu$$, but I would have expected

\begin{align} p(D' \mid D) &= \int \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) d \mu \\ &= \int \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) d \mu \\ &= \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) \int d \mu \end{align}

But it's unclear what becomes of the integral. It's not a probability, so it's not like this is guaranteed to be 1.

### Edit

This is my derivation based on Murphy's hint to look at Bishop (2006), page 93. Since both our posterior and prior are Gaussians, we can use the following fact:

\begin{align} p(\textbf{x}) &= \mathcal{N}(\textbf{x} \mid \boldsymbol{\mu}, \boldsymbol{\Psi}) \\ p(\textbf{y} \mid \textbf{x}) &= \mathcal{N}(\textbf{y} \mid A \textbf{x} + \textbf{b}, \textbf{P}) \\ &\Downarrow \\ p(\textbf{y}) &= \mathcal{N}(\textbf{y} \mid \textbf{A} \boldsymbol{\mu} + \textbf{b}, \textbf{P} + \textbf{A} \boldsymbol{\Psi} \textbf{A}^{\top}) \end{align}

where we have

\begin{align} \textbf{x} &= \mu \\ \boldsymbol{\mu} &= \mu_N \\ \boldsymbol{\Psi} &= \sigma_N^2 \\ \textbf{y} &= D' \\ \textbf{A} &= 1 \\ \textbf{b} &= 0 \\ \textbf{P} &= \sigma^2 \end{align}

This gives us

\begin{align} p(\mu) &= \mathcal{N}(\mu \mid \mu_N, \mu_N^2) \\ p(D' \mid \mu) &= \mathcal{N}(D' \mid \mu, \sigma^2) \\ p(D' \mid \mu) p(\mu) = p(D') &= \mathcal{N}(D' \mid \mu_N, \sigma^2 + \sigma_N^2) \end{align}

We can add conditioning on $$D$$ at every step if we'd like, since it doesn't effect the distributions provided we have the parameters (i.i.d.):

\begin{align} p(\mu \mid D) &= \mathcal{N}(\mu \mid \mu_N, \mu_N^2) \\ p(D' \mid \mu, D) &= \mathcal{N}(D' \mid \mu, \sigma^2) \\ p(D' \mid \mu, D) p(\mu) = p(D' \mid D) &= \mathcal{N}(D' \mid \mu_N, \sigma^2 + \sigma_N^2) \end{align}

• How did you manage to show that $\mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2)$ without integrating out $\mu$, after which the integral has disappeared? Apr 11 '19 at 20:58
• As far as I can tell, $p(D^\prime\mid D)$ isn't a probability, either: it's a probability density and therefore can have any nonnegative value--and even be infinite.
– whuber
Apr 11 '19 at 21:17
• @jbowman, I've added the derivation I used based on Murphy's hint to look at Bishop, equation 2.115.
– gwg
Apr 11 '19 at 21:27
• The line after "I would have expected" doesn't make sense; you are integrating with respect to $u$, but there is no $u$ in the formula being integrated. Do you mean to integrate wrt $\mu$? If so, note that $\mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2)$ is not true; what is true is that $\int_{\mu} \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2)d\mu = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2)$, so your next derivation won't have a $du$ in it if you do it correctly. Apr 11 '19 at 21:56
• Thanks for your help. Yes, I meant w.r.t $\mu$ (so $d \mu$) and have fixed that typo. I appreciate your comment, but my entire question is why your comment holds. Another way of asking my question is: what is wrong with my derivation in my edit? In my edit, I show my logic for why $\mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2)$. I understand that this must be wrong (or is inconsistent with Murphy). I don't understand why it is wrong.
– gwg
Apr 11 '19 at 22:06

If I understand correctly the question, it seems to me (and others before me in the comment section) that the derivation $$\mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2)$$ or equivalently $$p(D' \mid \mu, D) p(\mu) = p(D' \mid D) = \mathcal{N}(D' \mid \mu_N, \sigma^2 + \sigma_N^2)$$ is incorrect since the left hand side is a joint density on $$(D',\mu)$$ and the right hand side is a marginal density on $$D'$$. (When removing $$\mu$$ from the above rhs, it is as if $$\mu$$ is already integrated, but I advise against such loose reasonging.)
What is correct is that, if $$\underbrace{D'|\mu\sim}_\text{conditional}\mathcal{N}(\mu,\sigma^2)\qquad\text{and}\qquad\underbrace{\mu\sim}_\text{marginal}\mathcal{N}(\mu_N, \sigma_N^2)$$then $$\underbrace{D'\sim}_\text{marginal}\mathcal{N}(\mu_N, \sigma^2+\sigma_N^2)$$ as stated in the book.
• I see. I used Bishop's 2.115 incorrectly. It is not that $p(y) = p(y \mid x)p(x)$, which is obviously wrong now that I write it out explicitly. It is $p(y) = \int p(y \mid x)p(x) dx$.