3
$\begingroup$

I understand the prior and posterior distributions and I have read what the prior and posterior predictive distributions are.

However, I don't really see the point of knowing them.

Knowing more things wouldn't hurt, but I just want to understand the reason why I need to know them.

$\endgroup$
  • 4
    $\begingroup$ I see you've chosen the tag naive-bayes. Are you asking specifically in the context of a naive Bayes classifier or in general? $\endgroup$ – Glen_b Mar 8 '17 at 23:06
  • $\begingroup$ I didn't know the tag, naive-bayes, is only for the naive Bayes classifier. It doesn't necessarily have to be a term of naive Bayes classifier, it can be a general term. It is because I have studies the prior and predictive distributions, but I just can't find a reason for knowing it. I just want to understand why I had to know them after having studied the meanings of prior and posterior distributions. $\endgroup$ – user122358 Mar 9 '17 at 2:22
  • 3
    $\begingroup$ If you have read about those terms, you must have read the surrounding material, which hopefully contained some indications as to why those functions are of importance. If you have not, I suggest you check in BDA. For instance, in Section 6.3 and on page 162. $\endgroup$ – Xi'an Mar 9 '17 at 10:35
  • 1
    $\begingroup$ @Glen_b This appears to be a naive question about Bayes. $\endgroup$ – Sycorax Mar 9 '17 at 15:48
  • $\begingroup$ I've interpreted your question to be what useful, practical purpose these distributions have to motivate understanding them, and have answered in that light. If I'm incorrect, do say so and I'll delete my answer. $\endgroup$ – Sean Easter Mar 9 '17 at 16:30
6
$\begingroup$

Let's you denote your data as $X$ and imagine that you have some probabilistic model that describes your data in terms of the likelihood of observing your data given some parameter $\theta$. The parameter $\theta$ is unknown and is to be estimated from your data. To estimate your parameter you could use many different approaches, e.g. use maximum likelihood estimation to find such value of $\theta$ that maximizes the likelihood, or use Bayesian approach. In Bayesian approach to estimate the parameter we need one more thing, a prior distribution for $\theta$. If you take those things together, you can use Bayes theorem to obtain the posterior distribution of $\theta$ (i.e. your estimate):

$$ \underbrace{p(\theta|X)}_\text{posterior} \propto \underbrace{p(X|\theta)}_\text{likelihood} \, \underbrace{p(\theta)}_\text{prior} $$

So to catch up:

  • prior is the distribution of $\theta$ that is assumed by you before seeing the data. You do not "know" it, it is something that you assume.
  • likelihood is the conditional distribution of data given the prior, it defines your model,
  • posterior is the "estimated" distribution of parameter $\theta$ given data and the prior,
  • posterior predictive distribution is the distribution of data that is "predicted" by your model given your data and the prior, it describes your predictions from the model.

For example, in the classical beta-binomial model (see for multiple examples and more details) we have

$$ X \mid \theta \sim \mathrm{Binom}(n, \theta) \\ \theta \sim \mathrm{Beta}(\alpha, \beta) $$

so binomial distribution parametrized by $\theta$ is our likelihood, beta distribution with hyperparameters $\alpha,\beta$ is our prior and by using conjugacy we can obtain a closed-form solution for posterior

$$ \theta \mid X \sim \mathrm{Beta}(x+\alpha, n-x+\beta) $$

and posterior predictive distribution

$$ \tilde X \mid X,\theta \sim \mathrm{BetaBinom}(n, x+\alpha, n-x+\beta) $$

where $\tilde X$ are the $X$'s predicted by our model.

If this all is still unclear for you you can check the multiple threads tagged as on this site, or multiple handbooks on this topic.

$\endgroup$
6
$\begingroup$

Some uses of the posterior predictive:

  • Simulating future data based on your model assumptions and data observed to this point. This is useful for predictions, forecasting, etc.
  • Model checking via posterior predictive checking. Some comments have directed you to Bayesian Data Analysis, and its author has made a relevant chapter available. Tim's answer to this question should also prove helpful.

I've less help to offer on the prior predictive. I've found it useful as a sort of summary check on my combined priors: It can serve as an intuitive summary of your ultimate prior assumptions on expected data.

In a similar vein, some view it as a tool to arrive at informative priors. Consider this correspondence shared on Andrew Gelman's blog:

I don’t ever see parameters. Some model have few and some have hundreds. Instead, I see data. So I don’t know how to have an opinion on parameters themselves. Rather I think it far more natural to have opinions on the behavior of models. The prior predictive density is a good and sensible notion.

A further post continues:

The goal is to use the “black box” of the prior predictive density and the prior conditional density (the conditional in particular since you can look at model behaviour in a dynamic, scenario based setting) to inform us about how the informative priors should be constrained.

Put another way, if you're struggling to set prior parameters, you may find it sensible to examine those parameters' consequences on expected data. Doing so requires the prior predictive.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.