9
$\begingroup$

Mathematically, it's often seen that expressions and algorithms for Expectation Maximization (EM) are often simpler for mixed models, yet it seems that almost everything (if not everything) that can be solved with EM can also be solved with MLE (by, say, the Newton-Raphson method, for expressions that are not closed).

In literature, though, it seems that many favour EM over other methods (including minimization of the LL by, say, gradient descent); is it because of its simplicity in these models? Or is it for other reasons?

$\endgroup$
14
$\begingroup$

I think there's some crossed wires here. The MLE, as referred to in the statistical literature, is the Maximum Likelihood Estimate. This is an estimator. The EM algorithm is, as the name implies, an algorithm which is often used to compute the MLE. These are apples and oranges.

When the MLE is not in closed form, a commonly used algorithm for finding this is the Newton-Raphson algorithm, which may be what you are referring to when you state "can also be solved with MLE". In many problems, this algorithm works great; for "vanilla" problems, it's typically hard to beat.

However, there are plenty of problems where it fails, such as mixture models. My experience with various computational problems has been that while the EM algorithm is not always the fastest choice, it's often the easiest for a variety of reasons. Many times with novel models, the first algorithm used to find the MLE will be an EM algorithm. Then, several years later, researchers may find that a significantly more complicated algorithm is significantly faster. But these algorithms are non-trival.

Additionally, I speculate that much of the popularity of the EM-algorithm is the statistical flavor of it, helping statisticians feel differentiated from numerical analysts.

$\endgroup$
  • 3
    $\begingroup$ "...helping statisticians feel differentiated from numerical analysts"---I will definitely save this line for later use. $\endgroup$ – Guillermo Angeris Jun 26 '15 at 22:37
  • $\begingroup$ Additionally (I just updated the question, because it was my original intent to also include this), but why should we use EM instead of an algorithm like Gradient Descent? What's the preference for one to the other? Convergence speed, perhaps? $\endgroup$ – Guillermo Angeris Jun 26 '15 at 22:56
  • 1
    $\begingroup$ In the work that I've done, the biggest advantage of the EM-algorithm is the fact that the proposed parameter values are always valid: i.e. probability masses between [0,1] that sum to 1, which is not necessarily the case for gradient descent. Another advantage are that you should not have to calculate the likelihood to insure it has increased at every step. This is a big deal if the update can be calculated quickly, but the likelihood cannot. $\endgroup$ – Cliff AB Jun 26 '15 at 23:03
  • 3
    $\begingroup$ Another very nice aspect of the EM algorithm: tends to be much more numerically stable than gradient based methods. My research started with EM algorithms and it took me 4 years to realize how annoying numerical instability could be (i.e. when I started using non-EM algorithms). $\endgroup$ – Cliff AB Jun 27 '15 at 19:41
  • $\begingroup$ interesting. I guess this question just came up again for me, but what about doing something similar to convex optimization (on the sub-gradients) where you essentially perform gradient descent and then just project on the feasible set? I mean, it certainly sounds a lot harder than EM, but what would be some other downsides? $\endgroup$ – Guillermo Angeris Mar 23 '16 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.