I do not understand how one can prove that in a normal model, where we know the variance, but not the mean, the posterior predictive distribution is also normal...

Below is cut of a page from Gelman's Data Bayesian Analysis. I don't understand how the authors can state that the posterior predictive dist. will be a Normal one, i.e., why can they conclude that it's a joint normal posterior distribution and then conclude that the marginal is also normal?

enter image description here

Any help would be appreciated.


if $X \sim N(\mu_X, \sigma_X^2)$ and $Y \sim N(\mu_Y, \sigma_Y^2)$ and $X$ and $Y$ are independent then $$ f_Z(z) = \int_{-\infty}^\infty f_Y(z-x) f_X(x) dx $$ is a normal density with mean $\mu_X+\mu_Y$ and covariance $\sigma^2_X+\sigma_Y^2$ (that is actually the density of $Z=X+Y$). Simple proof at https://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables#Proof_using_convolutions which states your result.

  • $\begingroup$ Thanks for your answer peuhp. Actually, I don't think there's need for a change of variable. In the picture above, the $\theta$ is your $x$. What do you reckon? $\endgroup$ – An old man in the sea. Mar 29 '16 at 15:09
  • $\begingroup$ Happy to help. You completely right. I correct this $\endgroup$ – peuhp Mar 29 '16 at 15:11

If the likelihood is normal, i.e. $y_i|\theta \sim N(\theta,\sigma_x^2)$ and the prior is normal, i.e. $\theta_{prior} \sim N(\theta_0, \sigma_{\theta}^2) $, then the posterior is also normal, $\theta \sim N(\mu_1, \tau_1^2)$. You can follow a proof here.

Now, the predictive distribution will also be normal:

$P(y'|y) = \int_{\theta}P(y', \theta |y)d\theta = \int_{\theta}P(y'|y,\theta)P(\theta|y)d\theta = \int_{\theta}P(y'|\theta)P(\theta|y)d\theta$

The first step is just treating the probability as a marginal probability over the parameter space. The 2nd step is just the conditional probability. The 3rd step is the assumption that once we know $\theta$, it's the only thing that influences the new observation, and any old observation is useless. What you end up with is the integral over all $\theta$ of the likelihood times the posterior.

You could calculate these integrals and see that you end up with a normal distribution. Or you can use a little trick, using convolutions (that peuhp hints of):

$\int_{\theta}P(y' = y_0|\theta)P(\theta|y)d\theta = \int_{\theta}P(y' = y_0 - \theta+\theta|\theta)P(\theta|y)d\theta = \int_{\theta}P(y' = y_1 + \theta|\theta)P(\theta|y)d\theta = \int_{\theta}P(y'-\theta = y_1|\theta)P(\theta|y)d\theta = \int_{\theta}P(y'-\theta|\theta)P(\theta|y)d\theta$

Here $y_0$ is the value that the random variable is equal to. I used variable change and called $y_1 = y_0-\theta$.

Note that what we end up with is a convolution of two functions which belong to two distributions: $\theta$ which we already know; and $y-\theta |\theta \sim N(0, \sigma_x^2)$. I.e. what we get is the sum of these two normal distributions, which is equal to:

$$ y'|y \sim N(\mu_1, \sigma_x^2 + \tau_1^2) $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.