Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have a set of observations X that I believe were generated by a mixture of several probability distributions (specifically, two von mises and one uniform). I'd like to find the maximum likelihood estimate of the parameters of these distributions and their mixing coefficients.
What are the advantages and disadvantages of using the EM algorithm to identify these parameters, versus plugging the likelihood function into a nonlinear programming solver using trust region based methods?
The EM algorithm is very popular for fitting mixture distributions.
In general, both approaches will converge to a local optimum. A common heuristic for both approaches is to repeat several times from different initialization points.
The nonlinear programming approach would need constraints to be specified. In higher dimensions, this could get complicated (e.g. positive semidefiniteness constraints for covariance matrices). But, it sounds like your distributions are 1d, which simplifies things somewhat. You'd still need to constrain mixture weights to be nonnegative and sum to one, and the widths of component distributions to be nonnegative. The EM algorithm will naturally produce valid parameters for the mixture distribution.
Optimization with trust region methods requires the gradient of the objective function and possibly the Hessian, depending on the method. Gradients of the constraints are required as well. Things will be much faster if you can provide a function to compute these. That would require deriving an expression for the gradient/Hessian, or using automatic differentiation. Otherwise, finite differencing is typically used, which scales poorly with the number of parameters. The EM algorithm doesn't require the gradient.
From an implementation standpoint, the EM algorithm is often described as being very simple, but plugging things into a standard optimization solver sounds even simpler. If gradients/Hessians have to be derived, it may put the two approaches on more even footing in this regard.
Because of the uniform distribution, it seems like your objective function might have some nasty behavior. For example, imagine sliding the uniform distribution to the side. The likelihood won't change at all (the derivative will be zero w.r.t. the location parameter), until a data point falls into or out of the support of the distribution. The likelihood will then abruptly jump to a new value, and the function will be nondifferentiable at this point. This kind of behavior doesn't play nicely with gradient-based optimization algorithms. I don't know how it would affect the EM algorithm.
The EM algorithm can be used in cases where some data values are missing, although this is less relevant in the 1d case.
Possibly of interest:
