13
$\begingroup$

MLE = Maximum Likelihood Estimation

MAP = Maximum a posteriori

MLE is intuitive/naive in that it starts only with the probability of observation given the parameter (i.e. the likelihood function) and tries to find the parameter best accords with the observation. But it take into no consideration the prior knowledge.

MAP seems more reasonable because it does take into consideration the prior knowledge through the Bayes rule.

Here is a related question, but the answer is not thorough. https://dsp.stackexchange.com/questions/13174/differences-using-maximum-likelihood-or-maximum-a-posteriori-for-deconvolution-d

So, I think MAP is much better. Is that right? And when should I use which?

$\endgroup$
18
$\begingroup$

If a prior probability is given as part of the problem setup, then use that information (i.e. use MAP). If no such prior information is given or assumed, then MAP is not possible, and MLE is a reasonable approach.

$\endgroup$
  • 8
    $\begingroup$ It is worth adding that MAP with flat priors is equivalent to using ML. $\endgroup$ – Tim Mar 17 '18 at 19:03
  • $\begingroup$ Also worth noting is that if you want a mathematically "convenient" prior, you can use a conjugate prior, if one exists for your situation. $\endgroup$ – bean Mar 18 '18 at 17:05
8
$\begingroup$

A Bayesian would agree with you, a frequentist would not. This is a matter of opinion, perspective, and philosophy. I think that it does a lot of harm to the statistics community to attempt to argue that one method is always better than the other. Many problems will have Bayesian and frequentist solutions that are similar so long as the Bayesian does not have too strong of a prior.

$\endgroup$
  • 7
    $\begingroup$ It is not simply a matter of opinion. There are definite situations where one estimator is better than the other. $\endgroup$ – Tom Minka Oct 3 '14 at 18:32
  • 2
    $\begingroup$ @TomMinka I never said that there aren't situations where one method is better than the other! I simply responded to the OP's general statements such as "MAP seems more reasonable." Such a statement is equivalent to a claim that Bayesian methods are always better, which is a statement you and I apparently both disagree with. $\endgroup$ – jsk Oct 11 '14 at 5:05
  • $\begingroup$ jok is right. The Bayesian and frequentist approaches are philosophically different. So a strict frequentist would find the Bayesian approach unacceptable. $\endgroup$ – Michael R. Chernick Mar 17 '18 at 23:49
2
$\begingroup$

Assuming you have accurate prior information, MAP is better if the problem has a zero-one loss function on the estimate. If the loss is not zero-one (and in many real-world problems it is not), then it can happen that the MLE achieves lower expected loss. In these cases, it would be better not to limit yourself to MAP and MLE as the only two options, since they are both suboptimal.

$\endgroup$
  • $\begingroup$ The MAP estimator if a parameter depends on the parametrization, whereas the "0-1" loss does not. 0-1 in quotes because by my reckoning all estimators will typically give a loss of 1 with probability 1, and any attempt to construct an approximation again introduces the parametrization problem $\endgroup$ – guy Oct 3 '14 at 18:52
  • 1
    $\begingroup$ In my view, the zero-one loss does depend on parameterization, so there is no inconsistency. $\endgroup$ – Tom Minka Oct 4 '14 at 11:12
0
$\begingroup$

Short answer by @bean explains it very well. However, I would like to point to the section 1.1 of the paper Gibbs Sampling for the uninitiated by Resnik and Hardisty which takes the matter to more depth. I am writing few lines from this paper with very slight modifications (This answers repeats few of things which OP knows for sake of completeness)

MLE

Formally MLE produces the choice (of model parameter) most likely to generated the observed data.

MAP

A MAP estimated is the choice that is most likely given the observed data. In contrast to MLE, MAP estimation applies Bayes's Rule, so that our estimate can take into account prior knowledge about what we expect our parameters to be in the form of a prior probability distribution.

Catch

MLE and MAP estimates are both giving us the best estimate, according to their respective de nitions of "best". But notice that using a single estimate -- whether it's MLE or MAP -- throws away information. In principle, parameter could have any value (from the domain); might we not get better estimates if we took the whole distribution into account, rather than just a single estimated value for parameter? If we do that, we're making use of all the information about parameter that we can wring from the observed data, X.

So with this catch, we might want to use none of them. Also, as already mentioned by bean and Tim, if you have to use one of them, use MAP if you got prior. If you do not have priors, MAP reduces to MLE. Conjugate priors will help to solve the problem analytically, otherwise use Gibbs Sampling.

$\endgroup$
-2
$\begingroup$

If the data is less and you have priors available - "GO FOR MAP". If you have a lot data, the MAP will converge to MLE. Thus in case of lot of data scenario it's always better to do MLE rather than MAP.

$\endgroup$
  • $\begingroup$ It isn't that simple. $\endgroup$ – Michael R. Chernick Mar 17 '18 at 18:16
  • $\begingroup$ @MichaelChernick I might be wrong. I read this in grad school. I request that you correct me where i went wrong. $\endgroup$ – Heisenbug Mar 17 '18 at 23:22
  • $\begingroup$ The frequentist approach and the Bayesian approach are philosophically different. The frequency approach estimates the value of model parameters based on repeated sampling. The Bayesian approach treats the parameter as a random variable. So in the Bayesian approach you derive the posterior distribution of the parameter combining a prior distribution with the data. MAP looks for the highest peak of the posterior distribution while MLE estimates the parameter by only looking at the likelihood function of the data. $\endgroup$ – Michael R. Chernick Mar 17 '18 at 23:43
  • $\begingroup$ @MichaelChernick - Thank you for your input. But doesn't MAP behave like an MLE once we have suffcient data. If we break the MAP expression we get an MLE term also. With large amount of data the MLE term in the MAP takes over the prior. $\endgroup$ – Heisenbug Mar 18 '18 at 17:14
  • $\begingroup$ It depends on the prior and the amount of data. They can give similar results in large samples. The difference is in the interpretation. My comment was meant to show that it is not as simple as you make it. With a small amount of data it is not simply a matter of picking MAP if you have a prior. A poorly chosen prior can lead to getting a poor posterior distribution and hence a poor MAP. $\endgroup$ – Michael R. Chernick Mar 18 '18 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.