6
$\begingroup$

Let I have a Gaussian mixture consisting of $n$ Gaussians that is already fitted (e.g. using EM algorithm) with respect to a given data set. Now I want to add one more Gaussian to make the mixture more accurate.

Of course, I can start from scratch and run the same learning procedure for $n+1$ Gaussians. However, would it be possible to reuse the result from learning of $n$ Gaussians, e.g. for initialization?

I know that the $n$ Gaussians can be used somehow to form the prior for the $n+1$.

The accepted answer will contain:

  • a clear way (algorithm) how to use $n$ Gaussians for learning of $n+1$ Gaussians
  • some justification (explicitly stated or based on an authoritative reference) why the proposed algorithm is good.
$\endgroup$
  • 1
    $\begingroup$ Pardon my ignorance but why wouldn't you just fit n and n+1 models using EM and compare their fit? Given that your are interested in the n+1 solution, why give preference to the model with n Gaussians? Or why not start with the n+1 solution and remove a Gaussian? I don't offer these as critiques of the question - I am just trying to better understand it, as well as the responses below. $\endgroup$ – D L Dahly Oct 14 '14 at 14:53
  • $\begingroup$ The challenge is that in the considered application to fit 1,2,3... Gaussians seems to inefficient since the more from n to n+1 some information is lost. I do not start from n+1 since the time available is limited. Thus, I start from 1 Gaussian so I am all the time sure that I get all the time at least something. On the other hand I assume that more Gaussians can lead to better precision. From these reasons, it is highly desired to use the information from a simple model to fit a more complex one. Of course, there are numerous ways how to do it. $\endgroup$ – Karel Macek Oct 14 '14 at 18:09
  • 1
    $\begingroup$ This is a strange question in that a mixture with $n+1$ components has no reason to grow from a mixture with $n$ components by adding a component to the existing ones. To wit, RJMCMC algorithms that only use birth and death moves are known to be quite inefficient. $\endgroup$ – Xi'an Dec 20 '14 at 21:59
17
+100
$\begingroup$

If your goal is to find the maximum-likelihood mixture of size $n+1$, then you can use the existing solution as an initialization, once you have enlarged it to have one more Gaussian. To enlarge it, there are two approaches in the literature. The first approach is to add a new Gaussian in the best possible place, holding the existing ones fixed. This approach is described in Efficient Greedy Learning of Gaussian Mixture Models by Verbeek et al (2003). They give a fast heuristic for finding good parameters for the new Gaussian, and present experiments (but not theory) showing that it works well. In that paper, you can also find references to earlier work doing similar things.

The second approach is to split an existing Gaussian in two. This approach is described in SMEM Algorithm for Mixture Models by Ueda et al (2000), and Learning of Latent Class Models by Splitting and Merging Components by Karciauskas et al (2004). These papers were focused on splitting Gaussians as a way to escape local optima of the EM algorithm, but splitting can also be used for your situation.

$\endgroup$
  • $\begingroup$ Looks great, let me wait for other answers. This is a very promising one. My only question is: why these approaches make sense? Which one is better? $\endgroup$ – Karel Macek Oct 14 '14 at 18:10
7
$\begingroup$

Generally it depends on algorithm and software you use. In probably any software you can start your algorithm with some starting values.

However consider that you already fitted some model that is somehow "optimal" for the data. If you'll ask your algorithm for one more Gaussian than you already start in some local optima so it could get messy. It could be hard for the algorithm to find a "better" solution than the one that already is "the best of possible choices".

If you think of it in Bayesian way, you can consider it as if you were using some highly informative prior on your data and in this kind of cases sometimes it is hard for your data to overcome information coming from the prior.

Unfortunately I do not recall any literature that is dealing especially with this case. In his book Sivia (2006, Data Analysis: A Bayesian Tutorial. Oxford University Press.) argues that in Bayesian case using posterior from a previous analysis as a prior in a subsequent one is a bad idea in most cases.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.