2
$\begingroup$

I have some questions concerning estimating maximum likelihood of Gaussian mixture model.

  1. As I have read around some material, they usually use EM algorithm for maximizing the complete likelihood function (with hidden variables). If without introducing hidden variables, can I optimize directly the incomplete likelihood function? That is I consider $\Theta = \left( \pi, \theta \right)$ are the mixing proportion vector and the parameters vector, respectively, and then I compute the gradient of likelihood $\nabla L(\Theta)$ and run gradient methods.
  2. What is the benefits of introducing hidden variables and is there any reasons people tend to use EM over gradient methods?
$\endgroup$
1
$\begingroup$

Indeed, since the likelihood function $$\prod_{i=1}^n \sum_{j=1}^k p_j \varphi(x_i;\mu_j,\sigma_j)$$ is available in closed form, a regular maximisation algorithm like the gradient methods could be applied to this function. Unfortunately, they cannot work because the likelihood is not regular enough for them to find the global maximum:

  1. there are several equivalent local modes;
  2. it may happen that the likelihood is unbounded at the boundaries of the parameter space.

The EM algorithm also ends up in a local mode, hence shares some of these drawbacks but it is usually fast and can be repeated with different starting points to find a global mode.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.