Suppose you have a Normal Mixture Model with 2 Components - you could write this model as follows:

$\pi_1 N(\mu_1, \sigma_1) + \pi_2 N(\mu_2, \sigma_2)$

In the above model, there are 6 unknowns : $\pi_1$, $\mu_1$, $\sigma_1$, $\pi_2$, $\mu_2$, $\sigma_2$. In the context of Mixture Models, these 6 unknowns are generally estimated iteratively using the EM (Expectation Maximization) Algorithm.

My Question: What is preventing us from writing the above model in "Likelihood Format" as a function : $L(\pi_1, \mu_1, \sigma_1, \pi_2, \mu_2, \sigma_2) = \dots$ and then jointly estimating these 6 unknowns by either taking the partial derivatives of the likelihood function relative to each of these 6 terms and solving these equations - or using some optimization algorithm like Gradient Descent (if there is no closed form solution)?

Why has the EM algorithm become the standard algorithm for estimating the unknowns of Mixture Models?


  • 1
    $\begingroup$ There are five parameters as $\pi_2=1-\pi_1$ $\endgroup$
    – Xi'an
    May 6, 2022 at 6:05

1 Answer 1


While the observed likelihood is a well-defined function $$L(\theta|\mathbf x)=\prod_{i=1}^n \{\pi_1\varphi(x_i;\mu_1,\sigma_1)+ (1-\pi_1)\varphi(x_i;\mu_2,\sigma_2)\}$$ it does not offer enough regularity to apply an off-the-shelf optimisation algorithm like Newton or gradient, because it is a massively multimodal function, with further instable infinite modes on the boundary of the parameter space. EM offers the advantage of always converging to a local mode in a limited number of steps.

Redner and Walker (1984) propose a detailed theoretical analysis of the likelihood equations$$\nabla_\theta L(\theta|\mathbf x) = 0$$but the numerical issue of solving them and the inferential issue of selecting one of the solutions are pointed out as difficulties. See also this chapter by Cosma Shalizi (CMU).

  • $\begingroup$ Thank you for your answer! I am still trying to understand this fact - why would Newton/Gradient suffer from multimodal distributions but EM would not? And when you say convergence of EM, you are referring to "converging locally", or "converging globally"? Thanks! $\endgroup$
    – stats_noob
    May 6, 2022 at 16:19
  • $\begingroup$ local convergence $\endgroup$
    – Xi'an
    May 6, 2022 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.