I don't understand how when integrating over the parameters in the posterior predictive, the integration "disappears". It's hard for me to ask simply because I am confused, so here is an example.


Imagine we have a Gaussian model with unknown mean $\mu$ and fixed variance $\sigma^2$. If $D$ is the training data and $D'$ is unseen data, then the posterior predictive is

$$ \begin{align} p(D' \mid D) &= \int p(D' \mid D, \mu) p(\mu \mid D) d\mu \\ &\stackrel{\star}{=} \int p(D' \mid \mu) p(\mu \mid D) d\mu \\ &\triangleq \int \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) d \mu \end{align} $$

where step $\star$ holds because the modeling assumption is that $D'$ is conditionally independent from $D$ given $\mu$ and the definitions of $\mu_N$ and $\sigma_N^2$ fall out of computing the posterior. See Bishop (2006) page 98 for details.

Here is where I am confused. I can show that

$$ \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) $$

Murphy's derivation in Conjugate Bayesian analysis of the Gaussian distribution suggests looking at Bishop, equation 2.115 (see my edit for more). My trouble is, Murphy then claims that

$$ p(D \mid D') = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) $$

which is what I mean by the integration should "disappearing". What happened? I understand that this new distribution has no dependence on $\mu$, but I would have expected

$$ \begin{align} p(D' \mid D) &= \int \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) d \mu \\ &= \int \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) d \mu \\ &= \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) \int d \mu \end{align} $$

But it's unclear what becomes of the integral. It's not a probability, so it's not like this is guaranteed to be 1.


This is my derivation based on Murphy's hint to look at Bishop (2006), page 93. Since both our posterior and prior are Gaussians, we can use the following fact:

$$ \begin{align} p(\textbf{x}) &= \mathcal{N}(\textbf{x} \mid \boldsymbol{\mu}, \boldsymbol{\Psi}) \\ p(\textbf{y} \mid \textbf{x}) &= \mathcal{N}(\textbf{y} \mid A \textbf{x} + \textbf{b}, \textbf{P}) \\ &\Downarrow \\ p(\textbf{y}) &= \mathcal{N}(\textbf{y} \mid \textbf{A} \boldsymbol{\mu} + \textbf{b}, \textbf{P} + \textbf{A} \boldsymbol{\Psi} \textbf{A}^{\top}) \end{align} $$

where we have

$$ \begin{align} \textbf{x} &= \mu \\ \boldsymbol{\mu} &= \mu_N \\ \boldsymbol{\Psi} &= \sigma_N^2 \\ \textbf{y} &= D' \\ \textbf{A} &= 1 \\ \textbf{b} &= 0 \\ \textbf{P} &= \sigma^2 \end{align} $$

This gives us

$$ \begin{align} p(\mu) &= \mathcal{N}(\mu \mid \mu_N, \mu_N^2) \\ p(D' \mid \mu) &= \mathcal{N}(D' \mid \mu, \sigma^2) \\ p(D' \mid \mu) p(\mu) = p(D') &= \mathcal{N}(D' \mid \mu_N, \sigma^2 + \sigma_N^2) \end{align} $$

We can add conditioning on $D$ at every step if we'd like, since it doesn't effect the distributions provided we have the parameters (i.i.d.):

$$ \begin{align} p(\mu \mid D) &= \mathcal{N}(\mu \mid \mu_N, \mu_N^2) \\ p(D' \mid \mu, D) &= \mathcal{N}(D' \mid \mu, \sigma^2) \\ p(D' \mid \mu, D) p(\mu) = p(D' \mid D) &= \mathcal{N}(D' \mid \mu_N, \sigma^2 + \sigma_N^2) \end{align} $$

  • 2
    $\begingroup$ How did you manage to show that $ \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) $ without integrating out $\mu$, after which the integral has disappeared? $\endgroup$
    – jbowman
    Apr 11 '19 at 20:58
  • 1
    $\begingroup$ As far as I can tell, $p(D^\prime\mid D)$ isn't a probability, either: it's a probability density and therefore can have any nonnegative value--and even be infinite. $\endgroup$
    – whuber
    Apr 11 '19 at 21:17
  • $\begingroup$ @jbowman, I've added the derivation I used based on Murphy's hint to look at Bishop, equation 2.115. $\endgroup$
    – gwg
    Apr 11 '19 at 21:27
  • 1
    $\begingroup$ The line after "I would have expected" doesn't make sense; you are integrating with respect to $u$, but there is no $u$ in the formula being integrated. Do you mean to integrate wrt $\mu$? If so, note that $ \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) $ is not true; what is true is that $ \int_{\mu} \mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2)d\mu = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2) $, so your next derivation won't have a $du$ in it if you do it correctly. $\endgroup$
    – jbowman
    Apr 11 '19 at 21:56
  • $\begingroup$ Thanks for your help. Yes, I meant w.r.t $\mu$ (so $d \mu$) and have fixed that typo. I appreciate your comment, but my entire question is why your comment holds. Another way of asking my question is: what is wrong with my derivation in my edit? In my edit, I show my logic for why $\mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2)$. I understand that this must be wrong (or is inconsistent with Murphy). I don't understand why it is wrong. $\endgroup$
    – gwg
    Apr 11 '19 at 22:06

If I understand correctly the question, it seems to me (and others before me in the comment section) that the derivation $$\mathcal{N}(D' \mid \mu, \sigma^2) \mathcal{N}(\mu \mid \mu_N, \sigma_N^2) = \mathcal{N}(D' \mid \mu_N, \sigma_N^2 + \sigma^2)$$ or equivalently $$p(D' \mid \mu, D) p(\mu) = p(D' \mid D) = \mathcal{N}(D' \mid \mu_N, \sigma^2 + \sigma_N^2)$$ is incorrect since the left hand side is a joint density on $(D',\mu)$ and the right hand side is a marginal density on $D'$. (When removing $\mu$ from the above rhs, it is as if $\mu$ is already integrated, but I advise against such loose reasonging.)

What is correct is that, if $$\underbrace{D'|\mu\sim}_\text{conditional}\mathcal{N}(\mu,\sigma^2)\qquad\text{and}\qquad\underbrace{\mu\sim}_\text{marginal}\mathcal{N}(\mu_N, \sigma_N^2)$$then $$\underbrace{D'\sim}_\text{marginal}\mathcal{N}(\mu_N, \sigma^2+\sigma_N^2)$$ as stated in the book.

  • $\begingroup$ I see. I used Bishop's 2.115 incorrectly. It is not that $p(y) = p(y \mid x)p(x)$, which is obviously wrong now that I write it out explicitly. It is $p(y) = \int p(y \mid x)p(x) dx$. $\endgroup$
    – gwg
    Apr 15 '19 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.