MLE vs MAP estimation, when to use which?

MLE = Maximum Likelihood Estimation

MAP = Maximum a posteriori

MLE is intuitive/naive in that it starts only with the probability of observation given the parameter (i.e. the likelihood function) and tries to find the parameter best accords with the observation. But it take into no consideration the prior knowledge.

MAP seems more reasonable because it does take into consideration the prior knowledge through the Bayes rule.

Here is a related question, but the answer is not thorough.

So, I think MAP is much better. Is that right? And when should I use which?

If a prior probability is given as part of the problem setup, then use that information (i.e. use MAP). If no such prior information is given or assumed, then MAP is not possible, and MLE is a reasonable approach.

• It is worth adding that MAP with flat priors is equivalent to using ML.
– Tim
Mar 17, 2018 at 19:03
• Also worth noting is that if you want a mathematically "convenient" prior, you can use a conjugate prior, if one exists for your situation.
– bean
Mar 18, 2018 at 17:05
• Comment from @Tim should be the answer. Jun 29, 2021 at 10:09

A Bayesian would agree with you, a frequentist would not. This is a matter of opinion, perspective, and philosophy. I think that it does a lot of harm to the statistics community to attempt to argue that one method is always better than the other. Many problems will have Bayesian and frequentist solutions that are similar so long as the Bayesian does not have too strong of a prior.

• It is not simply a matter of opinion. There are definite situations where one estimator is better than the other. Oct 3, 2014 at 18:32
• @TomMinka I never said that there aren't situations where one method is better than the other! I simply responded to the OP's general statements such as "MAP seems more reasonable." Such a statement is equivalent to a claim that Bayesian methods are always better, which is a statement you and I apparently both disagree with.
– jsk
Oct 11, 2014 at 5:05
• jok is right. The Bayesian and frequentist approaches are philosophically different. So a strict frequentist would find the Bayesian approach unacceptable. Mar 17, 2018 at 23:49

Assuming you have accurate prior information, MAP is better if the problem has a zero-one loss function on the estimate. If the loss is not zero-one (and in many real-world problems it is not), then it can happen that the MLE achieves lower expected loss. In these cases, it would be better not to limit yourself to MAP and MLE as the only two options, since they are both suboptimal.

• The MAP estimator if a parameter depends on the parametrization, whereas the "0-1" loss does not. 0-1 in quotes because by my reckoning all estimators will typically give a loss of 1 with probability 1, and any attempt to construct an approximation again introduces the parametrization problem
– guy
Oct 3, 2014 at 18:52
• In my view, the zero-one loss does depend on parameterization, so there is no inconsistency. Oct 4, 2014 at 11:12

Short answer by @bean explains it very well. However, I would like to point to the section 1.1 of the paper Gibbs Sampling for the uninitiated by Resnik and Hardisty which takes the matter to more depth. I am writing few lines from this paper with very slight modifications (This answers repeats few of things which OP knows for sake of completeness)

MLE

Formally MLE produces the choice (of model parameter) most likely to generated the observed data.

MAP

A MAP estimated is the choice that is most likely given the observed data. In contrast to MLE, MAP estimation applies Bayes's Rule, so that our estimate can take into account prior knowledge about what we expect our parameters to be in the form of a prior probability distribution.

Catch

MLE and MAP estimates are both giving us the best estimate, according to their respective de nitions of "best". But notice that using a single estimate -- whether it's MLE or MAP -- throws away information. In principle, parameter could have any value (from the domain); might we not get better estimates if we took the whole distribution into account, rather than just a single estimated value for parameter? If we do that, we're making use of all the information about parameter that we can wring from the observed data, X.

So with this catch, we might want to use none of them. Also, as already mentioned by bean and Tim, if you have to use one of them, use MAP if you got prior. If you do not have priors, MAP reduces to MLE. Conjugate priors will help to solve the problem analytically, otherwise use Gibbs Sampling.

Theoretically, if you have the information about the prior probability, use MAP; otherwise MLE.

However, as the amount of data increases, the leading role of prior assumptions (which used by MAP) on model parameters will gradually weaken, while the data samples will greatly occupy a favorable position. In extreme cases, MLE is exactly same to MAP even if you remove the information about prior probability, i.e., assume the prior probability is uniformly distributed.

So:

• If dataset is large (like in machine learning): there is no difference between MLE and MAP; always use MLE.
• If dataset is small: MAP is much better than MLE; use MAP if you have information about prior probability.

As we know that

\begin{aligned} \hat\theta^{MAP}&=\arg \max\limits_{\substack{\theta}} \log P(\theta|\mathcal{D})\\ &= \arg \max\limits_{\substack{\theta}} \log \frac{P(\mathcal{D}|\theta)P(\theta)}{P(\mathcal{D})}\\ &=\arg \max\limits_{\substack{\theta}} \log P(\mathcal{D}|\theta)P(\theta) \\ &=\arg \max\limits_{\substack{\theta}} \underbrace{\log P(\mathcal{D}|\theta)}_{\text{log-likelihood}}+ \underbrace{\log P(\theta)}_{\text{regularizer}} \end{aligned}

The prior is treated as a regularizer and if you know the prior distribution, for example, Gaussin ($$\exp(-\frac{\lambda}{2}\theta^T\theta)$$) in linear regression, and it's better to add that regularization for better performance.

I think MAP is much better.

MAP is better compared to MLE, but here are some of its minuses:

1. It only provides a point estimate but no measure of uncertainty
2. It predicts overconfidently
3. Hard to summarize the posterior distribution, and the mode is sometimes untypical
4. Reparameterization invariance
5. The posterior cannot be used as the prior in the next step
6. Can’t compute credible intervals

If the data is less and you have priors available - "GO FOR MAP". If you have a lot data, the MAP will converge to MLE. Thus in case of lot of data scenario it's always better to do MLE rather than MAP.

• It isn't that simple. Mar 17, 2018 at 18:16
• @MichaelChernick I might be wrong. I read this in grad school. I request that you correct me where i went wrong. Mar 17, 2018 at 23:22
• The frequentist approach and the Bayesian approach are philosophically different. The frequency approach estimates the value of model parameters based on repeated sampling. The Bayesian approach treats the parameter as a random variable. So in the Bayesian approach you derive the posterior distribution of the parameter combining a prior distribution with the data. MAP looks for the highest peak of the posterior distribution while MLE estimates the parameter by only looking at the likelihood function of the data. Mar 17, 2018 at 23:43
• @MichaelChernick - Thank you for your input. But doesn't MAP behave like an MLE once we have suffcient data. If we break the MAP expression we get an MLE term also. With large amount of data the MLE term in the MAP takes over the prior. Mar 18, 2018 at 17:14
• It depends on the prior and the amount of data. They can give similar results in large samples. The difference is in the interpretation. My comment was meant to show that it is not as simple as you make it. With a small amount of data it is not simply a matter of picking MAP if you have a prior. A poorly chosen prior can lead to getting a poor posterior distribution and hence a poor MAP. Mar 18, 2018 at 17:49