7
$\begingroup$

As far as I know the usual method for estimating the parameters in GMM is EM. However, it is also possible to use maximum likelihood. What are the differences between these two methods? Why would one prefer either of them?

$\endgroup$
1
  • $\begingroup$ Why can't you directly compute the Maximum Likehood by setting the derivative to 0 and solving an equation system, but have to do it iteratively with EM? $\endgroup$ Oct 11, 2017 at 18:22

2 Answers 2

3
$\begingroup$

You can use ML directly but as the priors of the different gaussians (usually called latent variables) are unknown, you'll probably find that your optimization objective is pretty hard. EM iterative method solves this intractability.

Suggested read: https://see.stanford.edu/materials/aimlcs229/cs229-notes8.pdf

$\endgroup$
4
  • $\begingroup$ So, EM is almost always used in this case? $\endgroup$
    – Mehrin
    Oct 14, 2016 at 11:20
  • $\begingroup$ Yes, and as the number of gaussians in the mixture model grows, the probability of not using EM is reduced even further. $\endgroup$
    – Gabizon
    Oct 14, 2016 at 11:23
  • $\begingroup$ And do you know why gradient is used in Fisher vectors? Does it have to do with MLE? $\endgroup$
    – Mehrin
    Oct 14, 2016 at 11:27
  • $\begingroup$ This question is not related to your original question. Mark this question answered, ask another one and I'll answer there. $\endgroup$
    – Gabizon
    Oct 14, 2016 at 11:28
12
$\begingroup$

Mehrin, you are in danger of creating a false dichotomy. EM is an optimisation technique that can be used to find maximum likelihood estimates and so the choice is not "one or the other".

In mixture models EM is often used to find MLEs or MAPs as it produces transparent algorithms.

$\endgroup$
3
  • 2
    $\begingroup$ You can, however, use MLE directly i.e. take the gradient of pdf w.r.t. the mixture parameters and use gradient ascent. $\endgroup$
    – Mehrin
    Oct 14, 2016 at 10:46
  • 2
    $\begingroup$ @Mehrin, yes, but the question title should probably be changed to "direct optimization versus expectation-maximization algorithm." EM is an implementation of MLE. $\endgroup$
    – Adrian
    Oct 11, 2017 at 19:59
  • $\begingroup$ @Mehrin you could do this for the means/variances - but the cluster assignment probabilities phis are constrained to the simplex (sum to 1, all non negative). It is a constrained optimization, and gradient descent won't work on it. $\endgroup$ Nov 8, 2021 at 16:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.