Are MCMC based methods appropriate when Maximum a-posteriori estimation is available?

Question

I have been noticing that in many practical applications, MCMC-based methods are used to estimate a parameter even though the posterior is analytical (for example because the priors were conjugate). To me, it makes more sense to use of MAP-estimators rather than MCMC-based estimators. Could anyone point out why MCMC is still an appropriate method in the presence of an analytical posterior?

Answer 1

No need to use MCMC in this case: Markov Chain Monte-Carlo (MCMC) is a method used to generate values from a distribution. It produces a Markov chain of auto-correlated values with stationary distribution equal to the target distribution. This method will still work to get you what you want, even in cases where the target distribution has an analytic form. However, there are simpler and less computationally intensive methods that work in cases like this, where you are dealing with a posterior that has a nice analytic form.

In the case where the posterior distribution has an available analytic form, it is possible to obtain parameter estimates (e.g., MAP) by optimisation from that distribution using standard calculus techniques. If the target distribution is sufficiently simple you might get a closed form solution for the parameter estimator, but even if it is not, you can usually use simple iterative techniques (e.g., Newton-Raphson, gradient-descent, etc.) to find the optimising parameter estimate for any given input data. If you have an analytic form for the quantile function of the target distribution, and you need to generate values from the distribution, you can do this via inverse transform sampling, which is less computationally intensive than MCMC, and allows you to generate IID values rather than values with complex auto-correlation patterns.

In view of this, if you were programming from scratch, then there does not seem to be any reason you would use MCMC in the case where the target distribution has an available analytic form. The only reason you might do so is if you have a generic algorithm for MCMC already written, that can be implemented with minimal effort, and you decide that the efficiency of using the analytic form is outweighed by the effort to do the required math. In certain practical contexts you will be dealing with problems that are generally intractable, where MCMC algorithms are already set up and can be implemented with minimal effort (e.g., if you do data analysis in RStan). In these cases it may be easiest to run your existing MCMC methods rather than deriving analytic solutions to problems, though the latter can of course be used as a check on your working.

Answer 2

It is unclear to me what you call an analytical posterior $\pi(\theta)$ and hence why this analyticity should preclude one from using MCMC. Even for a posterior distribution that is available in closed form, including its normalising constant, which is how I understand analytical in this setting, there is no reason for Bayes estimates to be available in closed form, as solving the minimisation problem$$\min_\delta\int_\Theta \text{L}(\theta,\delta)\,\tilde\pi(\theta)\,f(x|\theta)\,\text{d}\theta$$when $\tilde\pi(\cdot)\propto\pi(\cdot)$ strongly depends on the loss function.

When the normalising constant $$\int \tilde\pi(\theta)\,\text{d}\theta$$is not available, finding a posterior mean or median or even mode [which does not require to know the constant], most often proceeds through an MCMC algorithm. For instance, if I am given the joint density, when $x,y\in(0,1)$, $$f_\theta(x,y)=\dfrac{1+\theta[(1+x)(1+y)-3]+\theta^2(1-x)(1-y)) }{[1-\theta(1-x)(1-y)]^3}\qquad\theta\in(-1,1)$$inspired by the Ali-Mikhail-Haq copula: it may be properly normalised (and is indeed), but the conditional expectation of $\Phi^{-1}(X)$ given $Y=y$ under this density, when $\Phi(.)$ is the Normal cdf, is not available in closed form. This is however a question of primary interest.

Note also that the maximum a posteriori estimator is not the most natural estimator in a Bayesian setting, since it does not correspond to a loss function and that closed-form representation of the density, even up to a constant, does not make finding the MAP necessarily easy. Or using the MAP relevant.

Answer 3

As I read it, this question is asking two somewhat orthogonal questions. One is should one use MAP-estimators over posterior means, and the other is whether one should MCMC if the posterior has an analytical form.

In regards to MAP estimators over posterior means, from a theoretical perspective, posterior means are generally preferred, as @Xian notes in his answer. The real advantage to MAP estimators is that, especially in the more typical case where the posterior is not in closed form, they can be calculated much faster (i.e. several orders of magnitude) than an estimate of the posterior mean. If the posterior is approximately symmetric (which often the case in many problems with large sample sizes), then MAP estimate should be very close to the posterior mean. So the attractiveness of the MAP is actually that it can be a very cheap approximation of the posterior mean.

Note that knowing the normalizing constant doesn't help us find the posterior mode, so having a closed form solution for the posterior technically doesn't help us find the MAP estimate, outside the case where we recognize the posterior as a specific distribution for which we know it's mode.

In regards to the second question, if one has a closed form the posterior distribution, generally speaking there's no reason to use MCMC algorithms. Theoretically, if you had a closed form solution for the posterior distribution, but didn't have a closed form for the mean of some function and couldn't take draws directly from this closed form distribution, then one might turn to MCMC algorithms. But I'm not aware of any cases of this situation.

Answer 4

I would argue that MCMC methods aren't necessarily inappropriate, even when closed-form solutions exist. Obviously, it's nice when an analytical solution exists: they are usually fast, you avoid concerns about convergence (etc).

On the other hand, consistency is also important. Switching from technique to technique complicates your presentation: at best, it's extraneous detail that may confuse or distract the audience away from your substantive result, and at worst it could look like an attempt at biasing the outcomes. If I had several models, only a few of which admit closed-form solutions, I would strongly consider running them all through the same MCMC pipeline even if it weren't strictly necessary.

I suspect this, plus inertia ("we have this script that works") accounts for most of what you're seeing.