I'm confused about the role of posterior predictive distributions in Bayesian inference and predictive inference.

As I understand it, the frequentist approach would typically involve fitting the MLE, which would return a single estimate of the parameter set for the data - so there is a single distribution for predicting new data points.

As I understand it, in Bayesian inference, the posterior predictive distribution is the probability distribution of a new data point, as opposed to the posterior distribution which is a distribution of the parameter. However, to obtain the posterior predictive distribution, you marginalise over the parameter by integrating over it.

$p(\tilde{x})|\alpha )=\int p(\tilde{x}|\theta ) p(\theta |\alpha )d\theta$

Each possible value of $\theta$ would give you a different distribution. Marginalising is (broadly) equivalent to taking a weighted average over the uncertainty in the parameter e.g. averaging over all the distributions that would arise from each possible value of the $\theta$, weighted by the probability of that $\theta$. If we imagine the levels within a Bayesian model (from $\alpha$ to $\theta$ to observations) then are we essentially averaging out over that middle layer?

If so, does this not also result in a single distribution for predicting new data points, just one that has accounted for the uncertainty while being determined? Does this not discard (or at least, use and then discard) useful information?

I am struggling to wrap my head around how this links to some other aspects/applications of Bayesian inference - which makes me suspect that I've misunderstood something.

  1. Prediction intervals: I thought that one of the key advantages of Bayesian inference is that enabled easier quantification of prediction intervals, which might be needed depending on how the prediction is being used (e.g. what if we are interested in the 95th or 99th quantile of the prediction interval). But by marginalising over $\theta$, have we not lost the information that would have enabled that? I have seen some discussion of posterior predictive intervals, but didn't really understand.
  2. Bayesian neural networks: Similar to the above, I thought that the key advantage of BNNs is that they allow for quantification of uncertainty. Would marginalising over the parameters not collapse this uncertainy?
  3. Heirarchical models: If you are trying to make prediction for a new, unknown, group within a heirarchical model, then it seems to me that you'd want to keep the quantified uncertainty (reflecting you don't know exactly the characteristics of this new group), which would exist in the variance of the second level of the model. Would marginalising for the posterior predictive require you to integrate away this (potentially insightful) uncertainty? Or do you only marginalise over the parameter uncertainty?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.