1
$\begingroup$

Let's assume that we generate some values by a mixture of two Gaussians. Now we want to find the parameters of the two Gaussians by likelihood maximisation. One good expect that the optimisation will find the two original Gaussians if amount of generated data is sufficient.

However, I observe the following behaviour. One of the Gaussians fits the two components of the distributions while the other one finds a single point and becomes narrower and narrower around this point.

Since the second Gaussian is very narrow (and becomes more and more narrower), the corresponding point get a huge probability density and the total likelihood goes to infinity.

So, obviously the procedure fails here. However, it is very surprising, since:

  1. I have a huge amount of the synthetic data.
  2. The model is very simple.
  3. The fitted model reflects the real generator perfectly (the data is generated by a mixture of two Gaussians).

So, my question is: Is the described problem well known and is there a standard solution to it?

$\endgroup$
2
$\begingroup$

Your Gaussian mixture model has encountered a singularity, which is a known problem when using a maximum likelihood approach to find the parameters of a GMM. Let's understand why this happens:

The log-likehood function of $K$-component GMM for some data $\textbf{X}$ is given by,

$$ \text{ln} \: p(\textbf{X} \mid \pi, \mu, \Sigma) = \sum_{n=1}^{N} \text{ln} \left\{\sum_{k=1}^{K}\pi_k \mathcal{N}(x_n \mid \mu_k, \Sigma_k)\right\} $$

When the mean of one of the components, $\mu_k$ equals an observation $x_n$, you get,

$$ \mathcal{N}(x_n \mid x_n, \sigma_j^2\textbf{I}) = \frac{1}{(2\pi)^{1/2}} \frac{1}{\sigma_j} $$

GMM Singularity

As variance $\sigma_j \rightarrow 0$, above term goes to infinity and drives the log-likelihood to infinity.

Ways to deal with singularities:

  • Take Bayesian approach
  • You can prevent the log-likelihood from blowing up, by detecting when a singularity occurs and avoid $x_n$ to become equal to $\mu_k$

Reference:

  • Pattern recognition and Machine Learning by Bishop
$\endgroup$
  • 1
    $\begingroup$ There should be a ln acting on the curly braces there. $\endgroup$ – Don Slowik May 20 at 15:07
  • 1
    $\begingroup$ @DonSlowik good catch! Fixed now. $\endgroup$ – kedarps May 20 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.