3
$\begingroup$

I have a set of observations X that I believe were generated by a mixture of several probability distributions (specifically, two von mises and one uniform). I'd like to find the maximum likelihood estimate of the parameters of these distributions and their mixing coefficients.

What are the advantages and disadvantages of using the EM algorithm to identify these parameters, versus plugging the likelihood function into a nonlinear programming solver using trust region based methods?

$\endgroup$

1 Answer 1

4
$\begingroup$

The EM algorithm is very popular for fitting mixture distributions.

In general, both approaches will converge to a local optimum. A common heuristic for both approaches is to repeat several times from different initialization points.

The nonlinear programming approach would need constraints to be specified. In higher dimensions, this could get complicated (e.g. positive semidefiniteness constraints for covariance matrices). But, it sounds like your distributions are 1d, which simplifies things somewhat. You'd still need to constrain mixture weights to be nonnegative and sum to one, and the widths of component distributions to be nonnegative. The EM algorithm will naturally produce valid parameters for the mixture distribution.

Optimization with trust region methods requires the gradient of the objective function and possibly the Hessian, depending on the method. Gradients of the constraints are required as well. Things will be much faster if you can provide a function to compute these. That would require deriving an expression for the gradient/Hessian, or using automatic differentiation. Otherwise, finite differencing is typically used, which scales poorly with the number of parameters. The EM algorithm doesn't require the gradient.

From an implementation standpoint, the EM algorithm is often described as being very simple, but plugging things into a standard optimization solver sounds even simpler. If gradients/Hessians have to be derived, it may put the two approaches on more even footing in this regard.

Because of the uniform distribution, it seems like your objective function might have some nasty behavior. For example, imagine sliding the uniform distribution to the side. The likelihood won't change at all (the derivative will be zero w.r.t. the location parameter), until a data point falls into or out of the support of the distribution. The likelihood will then abruptly jump to a new value, and the function will be nondifferentiable at this point. This kind of behavior doesn't play nicely with gradient-based optimization algorithms. I don't know how it would affect the EM algorithm.

The EM algorithm can be used in cases where some data values are missing, although this is less relevant in the 1d case.

Possibly of interest:

Why should one use EM vs. say, Gradient Descent with MLE?

What is the difference between EM and Gradient Ascent?

$\endgroup$
1
  • $\begingroup$ Why do you say that the EM algorithm doesn't require the gradient? In each maximisation step one needs to find the optimal parameters that maximise the expected log-likelihood. Wouldn't the gradient be helpful in this case? $\endgroup$
    – Alex
    Jul 31, 2018 at 0:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.