19
$\begingroup$

Mathematically, it's often seen that expressions and algorithms for Expectation Maximization (EM) are often simpler for mixed models, yet it seems that almost everything (if not everything) that can be solved with EM can also be solved with MLE (by, say, the Newton-Raphson method, for expressions that are not closed).

In literature, though, it seems that many favour EM over other methods (including minimization of the LL by, say, gradient descent); is it because of its simplicity in these models? Or is it for other reasons?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
22
$\begingroup$

I think there's some crossed wires here. The MLE, as referred to in the statistical literature, is the Maximum Likelihood Estimate. This is an estimator. The EM algorithm is, as the name implies, an algorithm which is often used to compute the MLE. These are apples and oranges.

When the MLE is not in closed form, a commonly used algorithm for finding this is the Newton-Raphson algorithm, which may be what you are referring to when you state "can also be solved with MLE". In many problems, this algorithm works great; for "vanilla" problems, it's typically hard to beat.

However, there are plenty of problems where it fails, such as mixture models. My experience with various computational problems has been that while the EM algorithm is not always the fastest choice, it's often the easiest for a variety of reasons. Many times with novel models, the first algorithm used to find the MLE will be an EM algorithm. Then, several years later, researchers may find that a significantly more complicated algorithm is significantly faster. But these algorithms are non-trival.

Additionally, I speculate that much of the popularity of the EM-algorithm is the statistical flavor of it, helping statisticians feel differentiated from numerical analysts.

Improve this answer
$\endgroup$
9
  • 4
    $\begingroup$ "...helping statisticians feel differentiated from numerical analysts"---I will definitely save this line for later use. $\endgroup$
    – Guillermo Angeris
    Jun 26, 2015 at 22:37
  • 1
    $\begingroup$ In the work that I've done, the biggest advantage of the EM-algorithm is the fact that the proposed parameter values are always valid: i.e. probability masses between [0,1] that sum to 1, which is not necessarily the case for gradient descent. Another advantage are that you should not have to calculate the likelihood to insure it has increased at every step. This is a big deal if the update can be calculated quickly, but the likelihood cannot. $\endgroup$
    – Cliff AB
    Jun 26, 2015 at 23:03
  • 4
    $\begingroup$ Another very nice aspect of the EM algorithm: tends to be much more numerically stable than gradient based methods. My research started with EM algorithms and it took me 4 years to realize how annoying numerical instability could be (i.e. when I started using non-EM algorithms). $\endgroup$
    – Cliff AB
    Jun 27, 2015 at 19:41
  • 1
    $\begingroup$ @GuillermoAngeris: I am guessing you are talking about the specific case of using the EM algorithm for mixture models (i.e. probability weights for gaussian mixtures, etc.). You could use constrained gradient based methods. As it turns out, I've actually write code which you could use to directly compare constrained gradient descent vs an EM based algorithm. It turns out that the EM algorithm dominates the constrained gradient descent algorithm (i.e. same cost per iteration, typically needs 1/100 as many iterations). $\endgroup$
    – Cliff AB
    Mar 24, 2016 at 3:15
  • 1
    $\begingroup$ @GuillermoAngeris: I've written up the little blog article about this. Take a look if you're interested! $\endgroup$
    – Cliff AB
    Mar 25, 2016 at 0:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.