To explain my question better, I will use this analogy: In the case of the Gradient-Descent method, we have multiple variations/expansions for the main algorithm, like stochastic gradient descent (SGD) or other variations like ADAM and so on and so forth.

When I came to the EM method, I could not find any variation/expansion for it like SGD (with metaparameters for example epoch and split size) to accelerate the method.

Is there any expansion or variation for the EM method?

  • 1
    $\begingroup$ There are tons, so it’s hard to give this question a complete answer. Note also that you can use the gradient descent variants you describe in the M-step, also. This would give you already several variants on EM. $\endgroup$ Jul 9 at 13:55
  • 2
    $\begingroup$ For a broad-brush overview, you might consider looking at Section 11.4.8: Other EM variants of Machine Learning: A Probabilistic Perspective (2012) by Murphy. A monograph length reference, The EM Algorithm and Extensions (2008) by McLahlan and Krishnan is also cited therein. $\endgroup$
    – microhaus
    Jul 9 at 14:09

Hope the following text from Wikipedia may dissipate your concerns:

A number of methods have been proposed to accelerate the sometimes slow convergence of the EM algorithm, such as those using conjugate gradient and modified Newton's methods (Newton–Raphson).[25] Also, EM can be used with constrained estimation methods.

Parameter-expanded expectation maximization (PX-EM) algorithm often provides speed up by "us[ing] a `covariance adjustment' to correct the analysis of the M step, capitalising on extra information captured in the imputed complete data".[26]

Expectation conditional maximization (ECM) replaces each M step with a sequence of conditional maximization (CM) steps in which each parameter θi is maximized individually, conditionally on the other parameters remaining fixed.[27] Itself can be extended into the Expectation conditional maximization either (ECME) algorithm.[28]

This idea is further extended in generalized expectation maximization (GEM) algorithm, in which is sought only an increase in the objective function F for both the E step and M step as described in the As a maximization–maximization procedure section.[15] GEM is further developed in a distributed environment and shows promising results.[29]

It is also possible to consider the EM algorithm as a subclass of the MM (Majorize/Minimize or Minorize/Maximize, depending on context) algorithm,[30] and therefore use any machinery developed in the more general case.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.