When doing maximum likelihood (ML) inference on a Gaussian mixture model (GMM), Bishop notes in PRML that if there is more than one mixture component in the GMM and the mean of one Gaussian collapses onto a data point, it is possible for the likelihood function to blow up to infinity as the variance of that Gaussian shrinks to zero. (This could happen, for example, when there is one "obvious" cluster and a single data point very far away, and there are two mixture components.) I understand that we are overfitting the training data by claiming the variance of one component is close to zero when it really may not be. But does doing ML on a GMM pose any theoretical issues that could prevent one from finding a solution? Isn't it still possible to find a solution using the EM algorithm?


First regarding the practical issue of vanishing variance most implementations of GMM will have a limit on how small the variance can get, you can set the minimum to some small value based on your specific problem.

Secondly EM is guaranteed to improve on the objective function (Likelihood in the case of ML) or remain the same. After each iteration the parameters are updated. If the parameters have converged you have a local optimum, which is your final solution. The short answer is there is nothing preventing you form finding such a solution - If you have the necessary precautions to prevent variance going to zero.

  • $\begingroup$ It's not very clear what you mean by saying "you have solutions at every iteration". I think I know what you're trying to say, but if I'm right, then this statement is not true. $\endgroup$
    – Cliff AB
    Jul 1 '15 at 17:18
  • $\begingroup$ Thanks, @adi. Can this be thought of as an identifiability problem different from the one typically discussed when talking about GMMs? That is, not only can we switch the labels of the data points around, but EM can also converge "early" if one of the cluster means collapses onto a point. This is because, the likelihood would go to infinity, causing the algorithm to converge. Depending on which mean collapses onto which point and how soon (in terms of EM iterations) convergence is reached (both of which depend on initialization), there could be many models with infinite likelihood. $\endgroup$ Jul 1 '15 at 17:30
  • $\begingroup$ @CliffAB I reworded my answer. I suppose you should call a configuration of parameters as a solution only if they are at an optima(?) $\endgroup$
    – A.D
    Jul 1 '15 at 17:35
  • $\begingroup$ @VivekSubramanian I would not consider variance going to zero (likelihood to inf) as converged, it is a degenerate solution. I do not think it is the same as the label identifiability issue - in that case you "actually" have a non-degenerate solution except the labels are reversed. $\endgroup$
    – A.D
    Jul 1 '15 at 17:40
  • 1
    $\begingroup$ @CliffAB - does my edit address your last point? $\endgroup$
    – A.D
    Jul 1 '15 at 17:46

In such a case (like many others : HMM, ME) you should be careful to use a logistic representation of you likelyhoods, ie store log(v) instead of v for a value v. The important point is to avoid floating point underflow/overflow.

  • $\begingroup$ Thanks, @mbl. Are you suggesting that the issues are only computational and not theoretical? $\endgroup$ Jul 1 '15 at 16:40
  • 3
    $\begingroup$ This answer seems to not related to the question - the question is not about numerical stability, but rather any theoretical reasons/scenarios where ML will not find a solution. $\endgroup$
    – A.D
    Jul 1 '15 at 17:01
  • $\begingroup$ "Logistic" does not mean "logarithmic," although the latter appears to be the intended term in this answer. $\endgroup$
    – whuber
    Oct 15 '15 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.