9
$\begingroup$

While studying Bayesian statistics, somehow I am facing a problem to understand the differences between prior distribution and prior predictive distribution. Prior distribution is sort of fine to understand but I have found it vague to understand the use of prior predictive distribution and why it is different from prior distribution.

$\endgroup$
10
$\begingroup$

Predictive here means predictive for observations. The prior distribution is a distribution for the parameters whereas the prior predictive distribution is a distribution for the observations.

If $X$ denotes the observations and we use the model (or likelihood) $p(x \mid \theta)$ for $\theta \in \Theta$ then a prior distribution is a distribution for $\theta$, for example $p_\beta(\theta)$ where $\beta$ is a set of hyperparameters. Note that there's no conditioning on $\beta$, and therefore the hyperparameters are considered fixed, which is not the case in hierarchical models but this not the point here.

The prior predictive distribution is the distribution of $X$ "averaged" over all possible values of $\theta$:

\begin{align*} p_\beta(x) &= \int_\Theta p(x , \theta) d\theta \\ &= \int_\Theta p(x \mid \theta) p_\beta(\theta) d\theta \end{align*}

This distribution is prior as it does not rely on any observation.

We can also define in the same way the posterior predictive distribution, that is if we have a sample $X = (X_1, \dots, X_n)$, the posterior predictive distribution is:

\begin{align*} p_\beta(x \mid X) &= \int_\Theta p(x ,\theta \mid X) d\theta \\ &= \int_\Theta p(x \mid \theta,X) p_\beta(\theta \mid X)d\theta \\ &= \int_\Theta p(x \mid \theta) p_\beta(\theta \mid X)d\theta. \end{align*} The last line is based on the assumption that the upcoming observation is independent of $X$ given $\theta$.

Thus the posterior predictive distribution is constructed the same way as the prior predictive distribution but while in the latter we weight with $p_\beta(\theta)$ in the former we weight with $p_\beta(\theta \mid X)$ that is with our "updated" knowledge about $\theta$.

Example : Beta-Binomial

Suppose our model is $X \mid \theta \sim {\rm Bin}(n,\theta)$ i.e $P(X = x \mid \theta) = \theta^x(1-\theta)^{n-x}$.
Here $\Theta = [0,1]$.

We also assume a beta prior distribution for $\theta$, $\beta(a,b)$, where $(a,b)$ is the set of hyper parameters.

The prior predictive distribution, $p_{a,b}(x)$, is the beta-binomial distribution with parameters $(n,a,b)$.

This discrete distribution gives the probability of getting $k$ successes out of $n$ trials given the hyper-parameters $(a,b)$ on the probability of success.

Now suppose we observe $n_1$ draws $(x_1, \dots, x_{n_1})$ with $m$ successes.

Since the binomial and beta distributions are conjugate distributions we have: \begin{align*} p(\theta \mid X=m) &\propto \theta^m (1 - \theta)^{n_1-m} \times \theta^{a-1}(1-\theta)^{b-1}\\ &\propto \theta^{a+m-1}(1-\theta)^{n_1+b-m-1} \\ &\propto \beta(a+m,n_1+b-m) \end{align*}

Thus $\theta \mid X$ follows a beta distribution with parameters $(a+m,n_1+b-m)$.

Then, $p_{a,b}(x \mid X = m)$ is also a beta-binomial distribution but this time with parameters $(n_2,a+m,b+n_1-m)$ rather than $(n_2,a,b)$.

Upon a $\beta(a,b)$ prior distribution and a ${\rm Bin}(n,\theta)$ likelihood, if we observe $m$ successes out of $n_1$ trials, the posterior predictive distribution is a beta-binomial with parameters $(n_2,a+x,b+n_1-x)$. Note that $n_2$ and $n_1$ play different roles here, since the posterior predictive distribution is about:

Given my current knowledge on $\theta$ after observing $m$ successes out of $n_1$ trials, i.e $\beta(n_1,a+x,n+b-x)$, what probability do I have of observing $k$ successes out of $n_2$ additional trials?

I hope this is useful and clear.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Yeap, I believe I have understood what you have explained here. Thank you very much. $\endgroup$ – Changhee Kang Feb 27 '19 at 13:30
  • $\begingroup$ good answer, but your integral expressions don't look right, and clearly have unstated assumptions about the prior/likelihood built in. The general integral is $p(x|X)=\int p(x,\theta|X)d\theta$ (ie integrate joint distribution to get marginal) $\endgroup$ – probabilityislogic Dec 9 '19 at 14:16
  • $\begingroup$ @probabilityislogic Thank you for pointing this out. I edited my answer, I hope this is better now. $\endgroup$ – winperikle Dec 9 '19 at 18:27
5
$\begingroup$

Let $Y$ be a random variable representing the (maybe future) data. We have a (parametric) model for $Y$ with $Y \sim f(y \mid \theta), \theta \in \Theta$, $\Theta$ the parameter space. Then we have a prior distribution represented by $\pi(\theta)$. Given an observation of $Y$, the posterior distribution of $\theta$ is $$ f(\theta \mid y) =\frac{f(y\mid\theta) \pi(\theta)}{\int_\Theta f(y\mid\theta) \pi(\theta)\; d\theta} $$ The prior predictive distribution of $Y$ is then the (modeled) distribution of $Y$ marginalized over the prior, that is, integrated over $\pi(\theta)$: $$ f(y) = \int_\Theta f(y\mid\theta) \pi(\theta)\; d\theta $$ that is, the denominator in Bayes theorem above. This is also called the preposterior distribution of $Y$. This tells you what data (that is $Y$) you expect to see before learning more about $\theta$. This have many uses, for instance in design of experiments, for an example, see Experimental Design on Testing Proportions or Intersections of chemistry and statistics.

Another use is as a way to understand the prior distribution better. Say you are interested in modeling the variation in weight of elephants, and your prior distribution leads to a prior predictive with substantial probability over 20 tons. Then you might want to rethink, typical weight of largest elephants is seldom above 6 tons, so a substantial probability over 20 tons seem wrong. One interesting paper in this direction is Gelman (which do not use the terminology ...)

Finally, preposterior concepts are typically not useful with uninformative priors, they require prior modeling taken serious. One example is the following: Let $Y \sim \mathcal{N}(\theta, 1)$ with a flat prior $\pi(\theta)=1$. Then the prior predictive of $Y$ is $$ f(y)= \int_{-\infty}^\infty \frac1{\sqrt{2\pi}} e^{-\frac12 (y-\theta)^2}\; d\theta = 1 $$ so is itself uniform, so not very useful.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.