6
$\begingroup$

While studying Bayesian statistics, somehow I am facing a problem to understand the differences between prior distribution and prior predictive distribution. Prior distribution is sort of fine to understand but I have found it vague to understand the use of prior predictive distribution and why it is different from prior distribution.

$\endgroup$
5
$\begingroup$

Predictive here means predictive for observations. The prior distribution is a distribution for the parameters whereas the prior predictive distribution is a distribution for the observation.

If $X$ denotes observation and we use the model (or likelihood) $p(x \mid \theta)$ then a prior distribution is a distribution for $\theta$, for example $p_\beta(\theta)$ where $\beta$ is a set of hyperparameters. Note that there's no conditioning on $\beta$ , and therefore the hyperparameters are considered fixed, which is not the case in hierarchical models but this not the point here.

The prior predictive distribution is the distribution of $X$ "averaged" over $\theta$,

$$ p_\beta(x) = \int p(x \mid \theta) p_\beta(\theta) d\theta $$

This distribution is prior as it does not rely on any observations.

We can also define the same way the posterior predictive distribution, that is if we have a sample $X = (X_1, \dots, X_n)$ the posterior predictive distribution is

\begin{align*} p_\beta(x \mid X) &= \int p(x \mid X, \theta) p_\beta(\theta) d\theta \\ &= \int p(x \mid \theta) p(X \mid \theta) p_\beta(\theta) d\theta \\ &= \int p(x \mid \theta) p_\beta(\theta \mid X)d\theta \end{align*}

thus the posterior predictive distribution is constructed the same way as the prior predictive distribution but while in the latter we weight with $p_\beta(\theta)$ is the former we weight with $p_\beta(\theta \mid X)$ that is with our "updated" knowledge about $\theta$.

Example : Beta-Binomial

Suppose our model is $X \mid \theta \sim Bin(n_1,\theta)$ i.e $P(X = x \mid \theta) = \theta^x(1-\theta)^{n_1-x}$.

We suppose a beta prior distribution for $\theta$, $\beta(a,b)$ where $(a,b)$ is the set of hyper parameters.

Then the prior predictive distribution for $\theta$ is the beta-binomial distribution of parameter $(n_1,a,b)$. This discrete distribution gives the probability of $k$ successes out of $n_1$ trials given hyper-parameter $(a,b)$ on the probability of success.

Now suppose we observe $n_1$ draws $(x_1, \dots, x_{n_1})$ whith $x$ successes.

Since the binomial and beta distributions are conjugate distributions we have: \begin{align*} p(\theta \mid X=x) &\propto \theta^x (1 - \theta)^{n_1-x} \times \theta^{a-1}(1-\theta)^{b-1}\\ &\propto \theta^{a+x-1}(1-\theta)^{n_1+b-x-1} \\ &\propto \beta(a+x,n_1+b-x) \end{align*}

Thus $\theta \mid x$ also follows a beta distribution. Then, $p(x \mid x, a,b)$ follows a beta-binomial but this time of parameters $(a+x,b+n_1-x)$ rather than $(a,b)$

Upon a $\beta(a,b)$ prior distribution and a $Bin(n_1,\theta)$ likelihood, if we observe $x$ successes out of $n_1$ trials the posterior predictive distribution is a beta-binomial of parameters $(n_2,a+x,b+n_1-x)$. Note that $n_2$ and $n_1$ play differents roles, since here the posterior predictive is about:

Given my current knowledge on $\theta$ after observing $x$ successes out of $n_1$ trials, i.e $\beta(n_1,a+x,n+b-x)$, what probability I have of observing $k$ successes out of $n_2$ additional trials.

I hope this is useful and clear

$\endgroup$
  • 1
    $\begingroup$ Yeap, I believe I have understood what you have explained here. Thank you very much. $\endgroup$ – Changhee Kang Feb 27 at 13:30
5
$\begingroup$

Let $Y$ be a random variable representing the (maybe future) data. We have a (parametric) model for $Y$ with $Y \sim f(y \mid \theta), \quad \theta \in \Theta$, $\Theta$ the parameter space. Then we have a prior distribution represented by $\pi(\theta)$. Given an observation of $Y$, the prior distribution of $\theta$ is $$ f(\theta \mid y) =\frac{f(y\mid\theta) \pi(\theta)}{\int_\Theta f(y\mid\theta) \pi(\theta)\; d\theta} $$ The prior predictive distribution of $Y$ is then the (modeled) distribution of $Y$ marginalized over the prior, that is, integrated over $\pi(\theta)$: $$ f(y) = \int_\Theta f(y\mid\theta) \pi(\theta)\; d\theta $$ that is, the denominator in Bayes theorem above. This is also called the preposterior distribution of $Y$. This tells you what data (that is $Y$) you expect to see before learning more about $\theta$. This have many uses, for instance in design of experiments, for an example, see Experimental Design on Testing Proportions or Intersections of chemistry and statistics.

Another use is as a way to understand the prior distribution better. Say you are interested in modeling the variation in weight of elephants, and your prior distribution leads to a prior predictive with substantial probability over 20 tons. Then you might want to rethink, typical weight of largest elephants is seldom above 6 tons, so a substantial probability over 20 tons seem wrong. One interesting paper in this direction is Gelman (which do not use the terminology ...)

Finally, preposterior concepts are typically not useful with uninformative priors, they require prior modeling taken serious. One example is the following: Let $Y \sim \mathcal{N}(\theta, 1)$ with a flat prior $\pi(\theta)=1$. Then the prior predictive of $Y$ is $$ f(y)= \int_{-\infty}^\infty \frac1{\sqrt{2\pi}} e^{-\frac12 (y-\theta)^2}\; d\theta = 1 $$ so is itself uniform, so not very useful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.