Posterior predictive distribution vs MAP estimate

Question

Consider a training dataset $X$, a probabilistic model parameterized by $\theta$, and a prior $P(\theta)$. For a new data point $x^*$, we can compute $P(x^*)$ using:

• a fully bayesian approach: the posterior predictive distribution $P(x^* | X) = \int P(\theta|X) P(x^*|\theta) d\theta$
• the likelihood parameterized by the maximum a posteriori estimate: $P(x^* | \theta_{MAP})$, where $\theta_{MAP} = \text{argmax}_\theta P(\theta|X)$

Is the fully bayesian approach always "better" than the MAP approach? More precisely, is the MAP approach an approximation of the bayesian approach, in the sense that we are we hoping that $P(x^* | \theta_{MAP})$ is a good approximation of $P(x^* | X)$?

Answer 1

I often think of it this way. In the fully Bayesian approach, we find the integral

$$p(x^*|X) = \int p(x^*|\theta) p(\theta|X) \text{ d}\theta$$

as integrating over all possible models (infinitely many in fact), and we make a prediction taking all of these models "into consideration". As this is often intractable, we use the MAP estimate of the posterior $p(\theta|X)$, which corresponds to evaluating the same integral but this time using a infinitely small part of $p(\theta|X)$, namely at its maximum. In other words, we multiply $p(x^*|\theta)$ with a new "delta-distribution" located at the max of the posterior distribution and integrate this to obtain the prediction.

The difference is therefore rather obvious: a fully Bayesian treatment corresponds to an infinite ensemble of models, where a given prediction $p(x|\textbf{x},\theta)$ is weighted by the model probability $p(\theta|\textbf{x})$, i.e. more likely models will contribute more to the prediction. The MAP estimate of the parameters will give you the prediction from one specific model, namely the most likely one according to Bayes theorem. Ensemble theory shows us that we often obtain better generalization and more accurate predictions and therefore this will often be "better" than the MAP.

Hope this helps.

Answer 2

Assuming your model is correctly specified, the predictive distribution gives an estimate of the new data point that takes account of all the uncertainty in the unknown parameter $\theta$. In the second method, where you merely use a parameter-substitution using your estimator, you are effectively treating this as a perfect estimator of the unknown parameter, and so the resulting "predictive" distribution does not take account of the uncertainty in the unknown parameter $\theta$. For this reason, the latter distribution will tend to have lower variability than the former, and if your model is correctly specified, this means that it underestimates the variability of the new data point. So yes, the predictive distribution is generally regarded as being "better".

Incidentally, this kind of comparison is not exclusive to Bayesian statistics. This methods you are comparing are very much like the analogous methods that occurs in frequentist methodology, where one can use a pivotal quantity to get a proper confidence interval for a new data point (analogous to a Bayesian predictive interval), or one can merely substitute the MLE as if it were a known parameter value and obtain an interval for a new data point from the sampling distribution (analogous to the Bayesian parameter-substitution method).