Estimating Mixture Models with Maximum Likelihood

Question

Suppose you have a Normal Mixture Model with 2 Components - you could write this model as follows:

$\pi_1 N(\mu_1, \sigma_1) + \pi_2 N(\mu_2, \sigma_2)$

In the above model, there are 6 unknowns : $\pi_1$, $\mu_1$, $\sigma_1$, $\pi_2$, $\mu_2$, $\sigma_2$. In the context of Mixture Models, these 6 unknowns are generally estimated iteratively using the EM (Expectation Maximization) Algorithm.

My Question: What is preventing us from writing the above model in "Likelihood Format" as a function : $L(\pi_1, \mu_1, \sigma_1, \pi_2, \mu_2, \sigma_2) = \dots$ and then jointly estimating these 6 unknowns by either taking the partial derivatives of the likelihood function relative to each of these 6 terms and solving these equations - or using some optimization algorithm like Gradient Descent (if there is no closed form solution)?

Why has the EM algorithm become the standard algorithm for estimating the unknowns of Mixture Models?

Thanks!

Answer 1

While the observed likelihood is a well-defined function $$L(\theta|\mathbf x)=\prod_{i=1}^n \{\pi_1\varphi(x_i;\mu_1,\sigma_1)+ (1-\pi_1)\varphi(x_i;\mu_2,\sigma_2)\}$$ it does not offer enough regularity to apply an off-the-shelf optimisation algorithm like Newton or gradient, because it is a massively multimodal function, with further instable infinite modes on the boundary of the parameter space. EM offers the advantage of always converging to a local mode in a limited number of steps.

Redner and Walker (1984) propose a detailed theoretical analysis of the likelihood equations$$\nabla_\theta L(\theta|\mathbf x) = 0$$but the numerical issue of solving them and the inferential issue of selecting one of the solutions are pointed out as difficulties. See also this chapter by Cosma Shalizi (CMU).