I have finally been able to wrap my head around the mechanics of how to initialize and train a multivariate Gaussian mixture model using expectation maximization algorithm. So I wonder how difficult this GMM and EM task is in comparison to all other common algorithms and models in machine learning. I appreciate any feedback. Thank you.

  • $\begingroup$ How do you mean difficult? Difficult to prepare the program? Difficult to find appropriate initial parameters? Difficult in terms of number of EM steps to reach the result? $\endgroup$
    – GaBorgulya
    Apr 2 '11 at 23:32
  • $\begingroup$ @GaBorgula: I mean the learning curve in general. $\endgroup$
    – Martin08
    Apr 3 '11 at 1:19

this paper is a small gem about fitting mixtures of gaussians to the data using Bayesian methods.

Big plus: no maximising algorithms required, so should be quick compared to EM algorithm. Also paper has testing data, so you can test each method against this one if you want. And it outlines a procedure for choosing the number of mixing components

The drawback is that it is based on a non-informative prior (so you must throw away info which isn't data), and a common variance (or kernel width).

  • $\begingroup$ The OP states they're interested in multivariate mixture modeling. The linked paper appears to treat only the univariate case. Is there an obvious extension to multivariate observations that remains clean, accessible, and for which the computation grows (reasonably) slowly with the dimensionality? $\endgroup$
    – cardinal
    Apr 3 '11 at 19:10
  • $\begingroup$ There is, but you would have to do the maths - I've had a go, and you get a mixture of multivariate student densities, but I can't exactly remember the formula, the digamma functions are a bit more tricky to approximate in the multivariate case. $\endgroup$ Apr 3 '11 at 22:46
  • $\begingroup$ Hmmm. If that's true, the approach of the paper sounds more appealing and appropriate as an alternative to univariate kernel density estimation than it does as an alternative to estimation using the EM algorithm for multivariate mixture modeling. $\endgroup$
    – cardinal
    Apr 4 '11 at 2:34
  • $\begingroup$ @cardinal - good point, but the multivariate generalisation is not polemic, just follows from using a multivariate guassian mixture model, and doing the same "splitting" of the data into "location learning" and "scale learning". Same with the choice of components. All the necessary principles of what is being done are there - Jose Bernardo just hasn't done the maths. $\endgroup$ Apr 4 '11 at 7:51

The log probability of a GMM is non-convex, which makes it converge only locally. Also, EM scales with the number of points in your dataset - you might want to try online EM if you have a big dataset.

Compared to fitting univariate models (like a single Gaussian) performance is of course horrible, since it's an iterative procedure. To speed it up, a good heuristic is to start of with a couple of iterations of K-Means, use the centers as means and go on from there with the GMM modeling.

I have had the feeling that GMMs converge globally for very simple and low dimensional datasets. I have never checked this for big datasets though.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.