What is the difference in Bayesian estimate and maximum likelihood estimate?

Question

Please explain to me the difference in Bayesian estimate and Maximum likelihood estimate?

Answer 1

It is a very broad question and my answer here only begins to scratch the surface a bit. I will use the Bayes's rule to explain the concepts.

Let’s assume that a set of probability distribution parameters, $\theta$, best explains the dataset $D$. We may wish to estimate the parameters $\theta$ with the help of the Bayes’ Rule:

$$p(\theta|D)=\frac{p(D|\theta) * p(\theta)}{p(D)}$$

$$posterior = \frac{likelihood * prior}{evidence}$$

The explanations follow:

Maximum Likelihood Estimate

With MLE,we seek a point value for $\theta$ which maximizes the likelihood, $p(D|\theta)$, shown in the equation(s) above. We can denote this value as $\hat{\theta}$. In MLE, $\hat{\theta}$ is a point estimate, not a random variable.

In other words, in the equation above, MLE treats the term $\frac{p(\theta)}{p(D)}$ as a constant and does NOT allow us to inject our prior beliefs, $p(\theta)$, about the likely values for $\theta$ in the estimation calculations.

Bayesian Estimate

Bayesian estimation, by contrast, fully calculates (or at times approximates) the posterior distribution $p(\theta|D)$. Bayesian inference treats $\theta$ as a random variable. In Bayesian estimation, we put in probability density functions and get out probability density functions, rather than a single point as in MLE.

Of all the $\theta$ values made possible by the output distribution $p(\theta|D)$, it is our job to select a value that we consider best in some sense. For example, we may choose the expected value of $\theta$ assuming its variance is small enough. The variance that we can calculate for the parameter $\theta$ from its posterior distribution allows us to express our confidence in any specific value we may use as an estimate. If the variance is too large, we may declare that there does not exist a good estimate for $\theta$.

As a trade-off, Bayesian estimation is made complex by the fact that we now have to deal with the denominator in the Bayes' rule, i.e. $evidence$. Here evidence -or probability of evidence- is represented by:

$$p(D) = \int_{\theta} p(D|\theta) * p(\theta) d\theta$$

This leads to the concept of 'conjugate priors' in Bayesian estimation. For a given likelihood function, if we have a choice regarding how we express our prior beliefs, we must use that form which allows us to carry out the integration shown above. The idea of conjugate priors and how they are practically implemented are explained quite well in this post by COOlSerdash.

Answer 2

I think you're talking about point estimation as in parametric inference, so that we can assume a parametric probability model for a data generating mechanism but the actual value of the parameter is unknown.

Maximum likelihood estimation refers to using a probability model for data and optimizing the joint likelihood function of the observed data over one or more parameters. It's therefore seen that the estimated parameters are most consistent with the observed data relative to any other parameter in the parameter space. Note such likelihood functions aren't necessarily viewed as being "conditional" upon the parameters since the parameters aren't random variables, hence it's somewhat more sophisticated to conceive of the likelihood of various outcomes comparing two different parameterizations. It turns out this is a philosophically sound approach.

Bayesian estimation is a bit more general because we're not necessarily maximizing the Bayesian analogue of the likelihood (the posterior density). However, the analogous type of estimation (or posterior mode estimation) is seen as maximizing the probability of the posterior parameter conditional upon the data. Usually, Bayes' estimates obtained in such a manner behave nearly exactly like those of ML. The key difference is that Bayes inference allows for an explicit method to incorporate prior information.

Also 'The Epic History of Maximum Likelihood makes for an illuminating read

http://arxiv.org/pdf/0804.2996.pdf

Answer 3

The Bayesian estimate is Bayesian inference while the MLE is a type of frequentist inference method.

According to the Bayesian inference, $f(x_1,...,x_n; \theta) = \frac{f(\theta; x_1,...,x_n) * f(x_1,...,x_n)}{f(\theta)}$ holds, that is $likelihood = \frac{posterior * evidence}{prior}$. Notice that the maximum likelihood estimate treats the ratio of evidence to prior as a constant(setting the prior distribution as uniform distribution/diffuse prior/uninformative prior, $p(\theta) = 1/6$ in playing a dice for instance), which omits the prior beliefs, thus MLE is considered to be a frequentist technique(rather than Bayesian). And the prior can be not the same in this scenario, because if the size of the sample is large enough MLE amounts to MAP(for detailed deduction please refer to this answer).

MLE's alternative in Bayesian inference is called maximum a posteriori estimation(MAP for short), and actually MLE is a special case of MAP where the prior is uniform, as we see above and as stated in Wikipedia:

From the point of view of Bayesian inference, MLE is a special case of maximum a posteriori estimation (MAP) that assumes a uniform prior distribution of the parameters.

For details please refer to this awesome article: MLE vs MAP: the connection between Maximum Likelihood and Maximum A Posteriori Estimation.

And one more difference is that maximum likelihood is overfitting-prone, but if you adopt the Bayesian approach the over-fitting problem can be avoided.

Answer 4

In principle the difference is precisely 0 - asymptotically speaking :)