MLE vs MAP estimation, when to use which?

Question

MLE = Maximum Likelihood Estimation

MAP = Maximum a posteriori

MLE is intuitive/naive in that it starts only with the probability of observation given the parameter (i.e. the likelihood function) and tries to find the parameter best accords with the observation. But it take into no consideration the prior knowledge.

MAP seems more reasonable because it does take into consideration the prior knowledge through the Bayes rule.

Here is a related question, but the answer is not thorough.

So, I think MAP is much better. Is that right? And when should I use which?

Answer 1

If a prior probability is given as part of the problem setup, then use that information (i.e. use MAP). If no such prior information is given or assumed, then MAP is not possible, and MLE is a reasonable approach.

Answer 2

A Bayesian would agree with you, a frequentist would not. This is a matter of opinion, perspective, and philosophy. I think that it does a lot of harm to the statistics community to attempt to argue that one method is always better than the other. Many problems will have Bayesian and frequentist solutions that are similar so long as the Bayesian does not have too strong of a prior.

Answer 3

Assuming you have accurate prior information, MAP is better if the problem has a zero-one loss function on the estimate. If the loss is not zero-one (and in many real-world problems it is not), then it can happen that the MLE achieves lower expected loss. In these cases, it would be better not to limit yourself to MAP and MLE as the only two options, since they are both suboptimal.

Answer 4

Short answer by @bean explains it very well. However, I would like to point to the section 1.1 of the paper Gibbs Sampling for the uninitiated by Resnik and Hardisty which takes the matter to more depth. I am writing few lines from this paper with very slight modifications (This answers repeats few of things which OP knows for sake of completeness)

MLE

Formally MLE produces the choice (of model parameter) most likely to generated the observed data.

MAP

A MAP estimated is the choice that is most likely given the observed data. In contrast to MLE, MAP estimation applies Bayes's Rule, so that our estimate can take into account prior knowledge about what we expect our parameters to be in the form of a prior probability distribution.

Catch

MLE and MAP estimates are both giving us the best estimate, according to their respective denitions of "best". But notice that using a single estimate -- whether it's MLE or MAP -- throws away information. In principle, parameter could have any value (from the domain); might we not get better estimates if we took the whole distribution into account, rather than just a single estimated value for parameter? If we do that, we're making use of all the information about parameter that we can wring from the observed data, X.

So with this catch, we might want to use none of them. Also, as already mentioned by bean and Tim, if you have to use one of them, use MAP if you got prior. If you do not have priors, MAP reduces to MLE. Conjugate priors will help to solve the problem analytically, otherwise use Gibbs Sampling.

Answer 5

Theoretically, if you have the information about the prior probability, use MAP; otherwise MLE.

However, as the amount of data increases, the leading role of prior assumptions (which used by MAP) on model parameters will gradually weaken, while the data samples will greatly occupy a favorable position. In extreme cases, MLE is exactly same to MAP even if you remove the information about prior probability, i.e., assume the prior probability is uniformly distributed.

So:

• If dataset is large (like in machine learning): there is no difference between MLE and MAP; always use MLE.
• If dataset is small: MAP is much better than MLE; use MAP if you have information about prior probability.

Answer 6

As we know that

$$\begin{equation}\begin{aligned} \hat\theta^{MAP}&=\arg \max\limits_{\substack{\theta}} \log P(\theta|\mathcal{D})\\ &= \arg \max\limits_{\substack{\theta}} \log \frac{P(\mathcal{D}|\theta)P(\theta)}{P(\mathcal{D})}\\ &=\arg \max\limits_{\substack{\theta}} \log P(\mathcal{D}|\theta)P(\theta) \\ &=\arg \max\limits_{\substack{\theta}} \underbrace{\log P(\mathcal{D}|\theta)}_{\text{log-likelihood}}+ \underbrace{\log P(\theta)}_{\text{regularizer}} \end{aligned}\end{equation}$$

The prior is treated as a regularizer and if you know the prior distribution, for example, Gaussin ($\exp(-\frac{\lambda}{2}\theta^T\theta)$) in linear regression, and it's better to add that regularization for better performance.

I think MAP is much better.

MAP is better compared to MLE, but here are some of its minuses:

1. It only provides a point estimate but no measure of uncertainty
2. It predicts overconfidently
3. Hard to summarize the posterior distribution, and the mode is sometimes untypical
4. Reparameterization invariance
5. The posterior cannot be used as the prior in the next step
6. Can’t compute credible intervals

Answer 7

If the data is less and you have priors available - "GO FOR MAP". If you have a lot data, the MAP will converge to MLE. Thus in case of lot of data scenario it's always better to do MLE rather than MAP.