I am a beginner in ML. I can do programming fine but the theory confuses me a lot of the times.

What is the relation between Max likelihood algorithm, Maximum A posteriori Method and Expectation Maximization method?

I see them used as the methods that actually do the optimization.

Please help in understanding them.

up vote 5 down vote accepted

Imagine that you have some data $X$ and probabilistic model parametrized by $\theta$, you are interested in learning about $\theta$ given your data. The relation between data, parameter and model is described using likelihood function

$$ \mathcal{L}(X \mid \theta) = p(\theta \mid X) $$

To find the best fitting $\theta$ you have to look for such value that maximizes the conditional probability of $\theta$ given $X$. Here things start to get complicated, because you can have different views on what $\theta$ is. You may consider it as a fixed parameter, or as a random variable. If you consider it as fixed, then to find it's value you need to find such value of $\theta$ that maximizes the likelihood function (maximum likelihood method [ML]). On another hand, if you consider it as a random variable, then this means that it also has some distribution, so you need to make one more assumption about prior distribution of $\theta$, i.e. $p(\theta)$, and you will be using Bayes theorem for estimation

$$ p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta) $$

If you are not interested in estimating the posterior distribution of $\theta$ but only about point estimate that maximizes the posterior probability, then you will be using maximum a posteriori (MAP) method for estimating it.

As about expectation-maximalization (EM), it is an algorithm that can be used in maximum likelihood approach for estimating certain kind of models (e.g. involving latent variables, or in missing data scenarios).

Check the following threads to learn more:
Maximum Likelihood Estimation (MLE) in layman terms
What is the difference between Maximum Likelihood Estimation & Gradient Descent?
Bayesian and frequentist reasoning in plain English
Who Are The Bayesians?
Is there a difference between the "maximum probability" and the "mode" of a parameter?
What is the difference between "likelihood" and "probability"?
Wikipedia entry on likelihood seems ambiguous
Numerical example to understand Expectation-Maximization

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.