When doing maximum likelihood (ML) inference on a Gaussian mixture model (GMM), Bishop notes in PRML that if there is more than one mixture component in the GMM and the mean of one Gaussian collapses onto a data point, it is possible for the likelihood function to blow up to infinity as the variance of that Gaussian shrinks to zero. (This could happen, for example, when there is one "obvious" cluster and a single data point very far away, and there are two mixture components.) I understand that we are overfitting the training data by claiming the variance of one component is close to zero when it really may not be. But does doing ML on a GMM pose any theoretical issues that could prevent one from finding a solution? Isn't it still possible to find a solution using the EM algorithm?
-
$\begingroup$ I don't think it'll completely answer your question, but some points on this topic can be found here: stats.stackexchange.com/questions/153254/… $\endgroup$– Cliff ABCommented Jul 1, 2015 at 17:15
2 Answers
First regarding the practical issue of vanishing variance most implementations of GMM will have a limit on how small the variance can get, you can set the minimum to some small value based on your specific problem.
Secondly EM is guaranteed to improve on the objective function (Likelihood in the case of ML) or remain the same. After each iteration the parameters are updated. If the parameters have converged you have a local optimum, which is your final solution. The short answer is there is nothing preventing you form finding such a solution - If you have the necessary precautions to prevent variance going to zero.
-
$\begingroup$ It's not very clear what you mean by saying "you have solutions at every iteration". I think I know what you're trying to say, but if I'm right, then this statement is not true. $\endgroup$– Cliff ABCommented Jul 1, 2015 at 17:18
-
$\begingroup$ Thanks, @adi. Can this be thought of as an identifiability problem different from the one typically discussed when talking about GMMs? That is, not only can we switch the labels of the data points around, but EM can also converge "early" if one of the cluster means collapses onto a point. This is because, the likelihood would go to infinity, causing the algorithm to converge. Depending on which mean collapses onto which point and how soon (in terms of EM iterations) convergence is reached (both of which depend on initialization), there could be many models with infinite likelihood. $\endgroup$ Commented Jul 1, 2015 at 17:30
-
1$\begingroup$ @VivekSubramanian I would not consider variance going to zero (likelihood to inf) as converged, it is a degenerate solution. I do not think it is the same as the label identifiability issue - in that case you "actually" have a non-degenerate solution except the labels are reversed. $\endgroup$– A.DCommented Jul 1, 2015 at 17:40
-
1$\begingroup$ The problem is that there is something from preventing you from finding a fixed point: if the algorithm begins to approach a degenerate solution (i.e. as solution where the variance parameter equals 0 for one or more Gaussian components), it will never converge. On the other hand, if it approaches a finite local maximum, it will converge. $\endgroup$– Cliff ABCommented Jul 1, 2015 at 17:41
-
1$\begingroup$ @CliffAB - does my edit address your last point? $\endgroup$– A.DCommented Jul 1, 2015 at 17:46
In such a case (like many others : HMM, ME) you should be careful to use a logistic representation of you likelyhoods, ie store log(v) instead of v for a value v. The important point is to avoid floating point underflow/overflow.
-
$\begingroup$ Thanks, @mbl. Are you suggesting that the issues are only computational and not theoretical? $\endgroup$ Commented Jul 1, 2015 at 16:40
-
3$\begingroup$ This answer seems to not related to the question - the question is not about numerical stability, but rather any theoretical reasons/scenarios where ML will not find a solution. $\endgroup$– A.DCommented Jul 1, 2015 at 17:01
-
$\begingroup$ "Logistic" does not mean "logarithmic," although the latter appears to be the intended term in this answer. $\endgroup$– whuber ♦Commented Oct 15, 2015 at 15:59