47
$\begingroup$

Please explain to me the difference in Bayesian estimate and Maximum likelihood estimate?

$\endgroup$
  • 7
    $\begingroup$ Depends on the kind of Bayesian estimate. MAP? Posterior mean? The result of minimizing Bayes risk for some loss function? Each of the above? Something else? $\endgroup$ – Glen_b Oct 29 '13 at 23:34
  • 2
    $\begingroup$ I've answered this question, or an analogue, here. stats.stackexchange.com/questions/73439/… What issues are you having understanding the two? More details will help us give a better answer. $\endgroup$ – Sycorax Oct 30 '13 at 12:15
  • 1
    $\begingroup$ From STAN reference manual: "If the prior is uniform, the posterior mode corresponds to the maximum likelihood estimate (MLE) of the parameters. If the prior is not uniform, the posterior mode is sometimes called the maximum a posterior (MAP) estimate." $\endgroup$ – Neerav Dec 22 '15 at 18:03
  • $\begingroup$ @Neerav that's the answer I needed. thx $\endgroup$ – javadba Nov 26 '17 at 1:33
  • $\begingroup$ A possibly useful answer for the specific case of Bayesian maximum a posteriori estimate is given here. $\endgroup$ – pglpm Mar 28 '18 at 13:24
65
$\begingroup$

It is a very broad question and my answer here only begins to scratch the surface a bit. I will use the Bayes's rule to explain the concepts.

Let’s assume that a set of probability distribution parameters, $\theta$, best explains the dataset $D$. We may wish to estimate the parameters $\theta$ with the help of the Bayes’ Rule:

$$p(\theta|D)=\frac{p(D|\theta) * p(\theta)}{p(D)}$$

$$posterior = \frac{likelihood * prior}{evidence}$$

The explanations follow:

Maximum Likelihood Estimate

With MLE,we seek a point value for $\theta$ which maximizes the likelihood, $p(D|\theta)$, shown in the equation(s) above. We can denote this value as $\hat{\theta}$. In MLE, $\hat{\theta}$ is a point estimate, not a random variable.

In other words, in the equation above, MLE treats the term $\frac{p(\theta)}{p(D)}$ as a constant and does NOT allow us to inject our prior beliefs, $p(\theta)$, about the likely values for $\theta$ in the estimation calculations.

Bayesian Estimate

Bayesian estimation, by contrast, fully calculates (or at times approximates) the posterior distribution $p(\theta|D)$. Bayesian inference treats $\theta$ as a random variable. In Bayesian estimation, we put in probability density functions and get out probability density functions, rather than a single point as in MLE.

Of all the $\theta$ values made possible by the output distribution $p(\theta|D)$, it is our job to select a value that we consider best in some sense. For example, we may choose the expected value of $\theta$ assuming its variance is small enough. The variance that we can calculate for the parameter $\theta$ from its posterior distribution allows us to express our confidence in any specific value we may use as an estimate. If the variance is too large, we may declare that there does not exist a good estimate for $\theta$.

As a trade-off, Bayesian estimation is made complex by the fact that we now have to deal with the denominator in the Bayes' rule, i.e. $evidence$. Here evidence -or probability of evidence- is represented by:

$$p(D) = \int_{\theta} p(D|\theta) * p(\theta) d\theta$$

This leads to the concept of 'conjugate priors' in Bayesian estimation. For a given likelihood function, if we have a choice regarding how we express our prior beliefs, we must use that form which allows us to carry out the integration shown above. The idea of conjugate priors and how they are practically implemented are explained quite well in this post by COOlSerdash.

$\endgroup$
  • 1
    $\begingroup$ Would you elaborate more on this? : "the denominator in the Bayes' rule, i.e. evidence." $\endgroup$ – Daniel Oct 30 '13 at 17:17
  • 1
    $\begingroup$ I extended my answer. $\endgroup$ – Zhubarb Oct 31 '13 at 9:41
  • $\begingroup$ @Berkan in the equation here, P(D|theta) is likelihood. However, likelihood function is defined as P(theta|D), that is the function of parameter, given data. I'm always confused about this. The term likelihood is refering to different things here? Could you elaborate on that? Thanks a lot! $\endgroup$ – zesla Jul 25 '18 at 19:46
  • 1
    $\begingroup$ @zesla if my understanding is correct, P(theta|D) is not the likelihood — it’s the posterior. That is, the distribution of theta conditional on the data source you have samples of. Likelihood is as you said: P(D|theta) — the distribution of your data as parameterized by theta, or put perhaps more intuitively, the “likelihood of seeing what you see” as a function of theta. Does that make sense? Everyone else: please correct me where I’m wrong. $\endgroup$ – grisaitis Aug 1 '18 at 4:54
  • $\begingroup$ @zesla, the explanation given by grisaitis is correct. $\endgroup$ – Zhubarb Aug 1 '18 at 7:57
13
$\begingroup$

I think you're talking about point estimation as in parametric inference, so that we can assume a parametric probability model for a data generating mechanism but the actual value of the parameter is unknown.

Maximum likelihood estimation refers to using a probability model for data and optimizing the joint likelihood function of the observed data over one or more parameters. It's therefore seen that the estimated parameters are most consistent with the observed data relative to any other parameter in the parameter space. Note such likelihood functions aren't necessarily viewed as being "conditional" upon the parameters since the parameters aren't random variables, hence it's somewhat more sophisticated to conceive of the likelihood of various outcomes comparing two different parameterizations. It turns out this is a philosophically sound approach.

Bayesian estimation is a bit more general because we're not necessarily maximizing the Bayesian analogue of the likelihood (the posterior density). However, the analogous type of estimation (or posterior mode estimation) is seen as maximizing the probability of the posterior parameter conditional upon the data. Usually, Bayes' estimates obtained in such a manner behave nearly exactly like those of ML. The key difference is that Bayes inference allows for an explicit method to incorporate prior information.

Also 'The Epic History of Maximum Likelihood makes for an illuminating read

http://arxiv.org/pdf/0804.2996.pdf

$\endgroup$
  • $\begingroup$ Would you elaborate more on this? "However, the analogous type of estimation (or posterior mode estimation) is seen as maximizing the probability of the posterior parameter conditional upon the data." $\endgroup$ – Daniel Oct 30 '13 at 17:14
  • $\begingroup$ The posterior mode is a bit of a misnomer because, with continuous DFs, the value is well defined. Posterior densities are related to the likelihood in the frequentist case, except that it allows you to simulate parameters from the posterior density. Interestingly, one most intuitively thinks of the "posterior mean" as being the best point estimate of the parameter. This approach is often done and, for symmetric unimodal densities, this produces valid credible intervals that are consistent with ML. The posterior mode is just the parameter value at the apex of the posterior density. $\endgroup$ – AdamO Oct 30 '13 at 17:29
  • $\begingroup$ About "this produces valid credible intervals that are consistent with ML.": It really depends on the model, right? They might be consistent or not ... $\endgroup$ – Daniel Oct 30 '13 at 18:19
  • 1
    $\begingroup$ The issue of underlying parametric assumptions motivates a discussion about fully parametric vs. semi-parametric or non-parametric inference. That is not a ML vs Bayesian issue and you're not the first to make that mistake. ML is a fully parametric approach, it allows you to estimate some things which SP or NP can't (and often more efficiently when they can). Correctly specifying the probability model in ML is exactly like choosing the correct prior and all the robustness properties (and sensitivity issues) that implies. $\endgroup$ – AdamO Oct 30 '13 at 18:55
  • $\begingroup$ BTW, your comments ignited this question in my mind. Any comments on this? stats.stackexchange.com/questions/74164/… $\endgroup$ – Daniel Oct 30 '13 at 19:14
2
$\begingroup$

Bayesian estimate is Bayesian inference while the MLE is a type of frequentist inference methods.

According to the Bayesian inference, $f(x_1,...,x_n; \theta) = \frac{f(\theta; x_1,...,x_n) * f(x_1,...,x_n)}{f(\theta)}$ holds, that is $likelihood = \frac{posterior * evidence}{prior}$. Notice that the maximum likelihood estimate treats the ratio of evidence to prior as a constant(setting the prior distribution as uniform distribution, $p(\theta) = 1/6$ in playing a dice for instance), which omits the prior beliefs, thus MLE is considered to be a frequentist technique(rather than Bayesian). And the prior can be not the same in this scenario, because if the samples are large enough MLE amounts to MAP(for detailed deduction please refer to this answer).

MLE's alternative in Bayesian inference is called maximum a posteriori estimation(MAP for short), and actually MLE is a special case of MAP where the prior is uniform, as we see above and as stated in Wikipedia:

From the point of view of Bayesian inference, MLE is a special case of maximum a posteriori estimation (MAP) that assumes a uniform prior distribution of the parameters.

For details please refer to this awesome article: MLE vs MAP: the connection between Maximum Likelihood and Maximum A Posteriori Estimation.

And one more difference is that maximum likelihood is overfitting-prone, but if you adopt the Bayesian approach the over-fitting problem can be avoided.

$\endgroup$
  • 1
    $\begingroup$ One of the cool things about Bayes is that you are not obligated to compute any point estimate at all. The entire posterior density can be your "estimate". $\endgroup$ – Frank Harrell Feb 20 '18 at 2:26
  • $\begingroup$ @FrankHarrell Dear Prof. Harrell, could you please help me edit the answer if I made some terrible mistakes somewhere? Thanks very much! $\endgroup$ – Lerner Zhang Feb 20 '18 at 2:35
  • 1
    $\begingroup$ I didn't mean to imply you had made a mistake. $\endgroup$ – Frank Harrell Feb 20 '18 at 4:54
  • $\begingroup$ @lerner: I want to warn against identifying the maximum-likelihood estimate as a particular case of the maximum-a-posteriori estimate (when the prior is constant): see why in this answer. $\endgroup$ – pglpm Mar 28 '18 at 12:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.