72
$\begingroup$

Please explain to me the difference in Bayesian estimate and Maximum likelihood estimate?

$\endgroup$
6
  • 10
    $\begingroup$ Depends on the kind of Bayesian estimate. MAP? Posterior mean? The result of minimizing Bayes risk for some loss function? Each of the above? Something else? $\endgroup$
    – Glen_b
    Commented Oct 29, 2013 at 23:34
  • 2
    $\begingroup$ I've answered this question, or an analogue, here. stats.stackexchange.com/questions/73439/… What issues are you having understanding the two? More details will help us give a better answer. $\endgroup$
    – Sycorax
    Commented Oct 30, 2013 at 12:15
  • 5
    $\begingroup$ From STAN reference manual: "If the prior is uniform, the posterior mode corresponds to the maximum likelihood estimate (MLE) of the parameters. If the prior is not uniform, the posterior mode is sometimes called the maximum a posterior (MAP) estimate." $\endgroup$
    – Neerav
    Commented Dec 22, 2015 at 18:03
  • $\begingroup$ @Neerav that's the answer I needed. thx $\endgroup$ Commented Nov 26, 2017 at 1:33
  • $\begingroup$ A possibly useful answer for the specific case of Bayesian maximum a posteriori estimate is given here. $\endgroup$
    – pglpm
    Commented Mar 28, 2018 at 13:24

4 Answers 4

92
$\begingroup$

It is a very broad question and my answer here only begins to scratch the surface a bit. I will use the Bayes's rule to explain the concepts.

Let’s assume that a set of probability distribution parameters, $\theta$, best explains the dataset $D$. We may wish to estimate the parameters $\theta$ with the help of the Bayes’ Rule:

$$p(\theta|D)=\frac{p(D|\theta) * p(\theta)}{p(D)}$$

$$posterior = \frac{likelihood * prior}{evidence}$$

The explanations follow:

Maximum Likelihood Estimate

With MLE,we seek a point value for $\theta$ which maximizes the likelihood, $p(D|\theta)$, shown in the equation(s) above. We can denote this value as $\hat{\theta}$. In MLE, $\hat{\theta}$ is a point estimate, not a random variable.

In other words, in the equation above, MLE treats the term $\frac{p(\theta)}{p(D)}$ as a constant and does NOT allow us to inject our prior beliefs, $p(\theta)$, about the likely values for $\theta$ in the estimation calculations.

Bayesian Estimate

Bayesian estimation, by contrast, fully calculates (or at times approximates) the posterior distribution $p(\theta|D)$. Bayesian inference treats $\theta$ as a random variable. In Bayesian estimation, we put in probability density functions and get out probability density functions, rather than a single point as in MLE.

Of all the $\theta$ values made possible by the output distribution $p(\theta|D)$, it is our job to select a value that we consider best in some sense. For example, we may choose the expected value of $\theta$ assuming its variance is small enough. The variance that we can calculate for the parameter $\theta$ from its posterior distribution allows us to express our confidence in any specific value we may use as an estimate. If the variance is too large, we may declare that there does not exist a good estimate for $\theta$.

As a trade-off, Bayesian estimation is made complex by the fact that we now have to deal with the denominator in the Bayes' rule, i.e. $evidence$. Here evidence -or probability of evidence- is represented by:

$$p(D) = \int_{\theta} p(D|\theta) * p(\theta) d\theta$$

This leads to the concept of 'conjugate priors' in Bayesian estimation. For a given likelihood function, if we have a choice regarding how we express our prior beliefs, we must use that form which allows us to carry out the integration shown above. The idea of conjugate priors and how they are practically implemented are explained quite well in this post by COOlSerdash.

$\endgroup$
7
  • 1
    $\begingroup$ Would you elaborate more on this? : "the denominator in the Bayes' rule, i.e. evidence." $\endgroup$
    – Daniel
    Commented Oct 30, 2013 at 17:17
  • 1
    $\begingroup$ I extended my answer. $\endgroup$
    – Zhubarb
    Commented Oct 31, 2013 at 9:41
  • 1
    $\begingroup$ @Berkan in the equation here, P(D|theta) is likelihood. However, likelihood function is defined as P(theta|D), that is the function of parameter, given data. I'm always confused about this. The term likelihood is refering to different things here? Could you elaborate on that? Thanks a lot! $\endgroup$
    – zesla
    Commented Jul 25, 2018 at 19:46
  • 2
    $\begingroup$ @zesla if my understanding is correct, P(theta|D) is not the likelihood — it’s the posterior. That is, the distribution of theta conditional on the data source you have samples of. Likelihood is as you said: P(D|theta) — the distribution of your data as parameterized by theta, or put perhaps more intuitively, the “likelihood of seeing what you see” as a function of theta. Does that make sense? Everyone else: please correct me where I’m wrong. $\endgroup$ Commented Aug 1, 2018 at 4:54
  • $\begingroup$ @zesla, the explanation given by grisaitis is correct. $\endgroup$
    – Zhubarb
    Commented Aug 1, 2018 at 7:57
16
$\begingroup$

I think you're talking about point estimation as in parametric inference, so that we can assume a parametric probability model for a data generating mechanism but the actual value of the parameter is unknown.

Maximum likelihood estimation refers to using a probability model for data and optimizing the joint likelihood function of the observed data over one or more parameters. It's therefore seen that the estimated parameters are most consistent with the observed data relative to any other parameter in the parameter space. Note such likelihood functions aren't necessarily viewed as being "conditional" upon the parameters since the parameters aren't random variables, hence it's somewhat more sophisticated to conceive of the likelihood of various outcomes comparing two different parameterizations. It turns out this is a philosophically sound approach.

Bayesian estimation is a bit more general because we're not necessarily maximizing the Bayesian analogue of the likelihood (the posterior density). However, the analogous type of estimation (or posterior mode estimation) is seen as maximizing the probability of the posterior parameter conditional upon the data. Usually, Bayes' estimates obtained in such a manner behave nearly exactly like those of ML. The key difference is that Bayes inference allows for an explicit method to incorporate prior information.

Also 'The Epic History of Maximum Likelihood makes for an illuminating read

http://arxiv.org/pdf/0804.2996.pdf

$\endgroup$
6
  • $\begingroup$ Would you elaborate more on this? "However, the analogous type of estimation (or posterior mode estimation) is seen as maximizing the probability of the posterior parameter conditional upon the data." $\endgroup$
    – Daniel
    Commented Oct 30, 2013 at 17:14
  • $\begingroup$ The posterior mode is a bit of a misnomer because, with continuous DFs, the value is well defined. Posterior densities are related to the likelihood in the frequentist case, except that it allows you to simulate parameters from the posterior density. Interestingly, one most intuitively thinks of the "posterior mean" as being the best point estimate of the parameter. This approach is often done and, for symmetric unimodal densities, this produces valid credible intervals that are consistent with ML. The posterior mode is just the parameter value at the apex of the posterior density. $\endgroup$
    – AdamO
    Commented Oct 30, 2013 at 17:29
  • $\begingroup$ About "this produces valid credible intervals that are consistent with ML.": It really depends on the model, right? They might be consistent or not ... $\endgroup$
    – Daniel
    Commented Oct 30, 2013 at 18:19
  • 1
    $\begingroup$ The issue of underlying parametric assumptions motivates a discussion about fully parametric vs. semi-parametric or non-parametric inference. That is not a ML vs Bayesian issue and you're not the first to make that mistake. ML is a fully parametric approach, it allows you to estimate some things which SP or NP can't (and often more efficiently when they can). Correctly specifying the probability model in ML is exactly like choosing the correct prior and all the robustness properties (and sensitivity issues) that implies. $\endgroup$
    – AdamO
    Commented Oct 30, 2013 at 18:55
  • $\begingroup$ BTW, your comments ignited this question in my mind. Any comments on this? stats.stackexchange.com/questions/74164/… $\endgroup$
    – Daniel
    Commented Oct 30, 2013 at 19:14
3
$\begingroup$

The Bayesian estimate is Bayesian inference while the MLE is a type of frequentist inference method.

According to the Bayesian inference, $f(x_1,...,x_n; \theta) = \frac{f(\theta; x_1,...,x_n) * f(x_1,...,x_n)}{f(\theta)}$ holds, that is $likelihood = \frac{posterior * evidence}{prior}$. Notice that the maximum likelihood estimate treats the ratio of evidence to prior as a constant(setting the prior distribution as uniform distribution/diffuse prior/uninformative prior, $p(\theta) = 1/6$ in playing a dice for instance), which omits the prior beliefs, thus MLE is considered to be a frequentist technique(rather than Bayesian). And the prior can be not the same in this scenario, because if the size of the sample is large enough MLE amounts to MAP(for detailed deduction please refer to this answer).

MLE's alternative in Bayesian inference is called maximum a posteriori estimation(MAP for short), and actually MLE is a special case of MAP where the prior is uniform, as we see above and as stated in Wikipedia:

From the point of view of Bayesian inference, MLE is a special case of maximum a posteriori estimation (MAP) that assumes a uniform prior distribution of the parameters.

For details please refer to this awesome article: MLE vs MAP: the connection between Maximum Likelihood and Maximum A Posteriori Estimation.

And one more difference is that maximum likelihood is overfitting-prone, but if you adopt the Bayesian approach the over-fitting problem can be avoided.

$\endgroup$
8
  • 4
    $\begingroup$ One of the cool things about Bayes is that you are not obligated to compute any point estimate at all. The entire posterior density can be your "estimate". $\endgroup$ Commented Feb 20, 2018 at 2:26
  • $\begingroup$ @FrankHarrell Dear Prof. Harrell, could you please help me edit the answer if I made some terrible mistakes somewhere? Thanks very much! $\endgroup$ Commented Feb 20, 2018 at 2:35
  • 1
    $\begingroup$ I didn't mean to imply you had made a mistake. $\endgroup$ Commented Feb 20, 2018 at 4:54
  • 1
    $\begingroup$ This is covered in multiple Bayesian texts. The end result is the posterior distribution. If you want to distill that to one point you need a loss function to reward or penalize certain properties of such point estimates. With squared error loss the best guess is the mean. With absolute error loss the best guess is the posterior median. $\endgroup$ Commented Apr 12, 2021 at 20:53
  • 1
    $\begingroup$ Yes, just keep in mind that we were talking about point estimates, not actual decisions. Bayesian decision making by optimizing expected utility does not involve any point estimates but instead uses only whole distributions. $\endgroup$ Commented Apr 13, 2021 at 20:06
0
$\begingroup$

In principle the difference is precisely 0 - asymptotically speaking :)

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.