Growing number of Gaussians in a mixture

Question

Let I have a Gaussian mixture consisting of $n$ Gaussians that is already fitted (e.g. using EM algorithm) with respect to a given data set. Now I want to add one more Gaussian to make the mixture more accurate.

Of course, I can start from scratch and run the same learning procedure for $n+1$ Gaussians. However, would it be possible to reuse the result from learning of $n$ Gaussians, e.g. for initialization?

I know that the $n$ Gaussians can be used somehow to form the prior for the $n+1$.

The accepted answer will contain:

• a clear way (algorithm) how to use $n$ Gaussians for learning of $n+1$ Gaussians
• some justification (explicitly stated or based on an authoritative reference) why the proposed algorithm is good.

Answer 1

If your goal is to find the maximum-likelihood mixture of size $n+1$, then you can use the existing solution as an initialization, once you have enlarged it to have one more Gaussian. To enlarge it, there are two approaches in the literature. The first approach is to add a new Gaussian in the best possible place, holding the existing ones fixed. This approach is described in Efficient Greedy Learning of Gaussian Mixture Models by Verbeek et al (2003). They give a fast heuristic for finding good parameters for the new Gaussian, and present experiments (but not theory) showing that it works well. In that paper, you can also find references to earlier work doing similar things.

The second approach is to split an existing Gaussian in two. This approach is described in SMEM Algorithm for Mixture Models by Ueda et al (2000), and Learning of Latent Class Models by Splitting and Merging Components by Karciauskas et al (2004). These papers were focused on splitting Gaussians as a way to escape local optima of the EM algorithm, but splitting can also be used for your situation.

Answer 2

Generally it depends on algorithm and software you use. In probably any software you can start your algorithm with some starting values.

However consider that you already fitted some model that is somehow "optimal" for the data. If you'll ask your algorithm for one more Gaussian than you already start in some local optima so it could get messy. It could be hard for the algorithm to find a "better" solution than the one that already is "the best of possible choices".

If you think of it in Bayesian way, you can consider it as if you were using some highly informative prior on your data and in this kind of cases sometimes it is hard for your data to overcome information coming from the prior.

Unfortunately I do not recall any literature that is dealing especially with this case. In his book Sivia (2006, Data Analysis: A Bayesian Tutorial. Oxford University Press.) argues that in Bayesian case using posterior from a previous analysis as a prior in a subsequent one is a bad idea in most cases.