I have finally been able to wrap my head around the mechanics of how to initialize and train a multivariate Gaussian mixture model using expectation maximization algorithm. So I wonder how difficult this GMM and EM task is in comparison to all other common algorithms and models in machine learning. I appreciate any feedback. Thank you.
this paper is a small gem about fitting mixtures of gaussians to the data using Bayesian methods.
Big plus: no maximising algorithms required, so should be quick compared to EM algorithm. Also paper has testing data, so you can test each method against this one if you want. And it outlines a procedure for choosing the number of mixing components
The drawback is that it is based on a non-informative prior (so you must throw away info which isn't data), and a common variance (or kernel width).
The log probability of a GMM is non-convex, which makes it converge only locally. Also, EM scales with the number of points in your dataset - you might want to try online EM if you have a big dataset.
Compared to fitting univariate models (like a single Gaussian) performance is of course horrible, since it's an iterative procedure. To speed it up, a good heuristic is to start of with a couple of iterations of K-Means, use the centers as means and go on from there with the GMM modeling.
I have had the feeling that GMMs converge globally for very simple and low dimensional datasets. I have never checked this for big datasets though.