When doing maximum likelihood (ML) inference on a Gaussian mixture model (GMM), Bishop notes in PRML that if there is more than one mixture component in the GMM and the mean of one Gaussian collapses onto a data point, it is possible for the likelihood function to blow up to infinity as the variance of that Gaussian shrinks to zero. (This could happen, for example, when there is one "obvious" cluster and a single data point very far away, and there are two mixture components.) I understand that we are overfitting the training data by claiming the variance of one component is close to zero when it really may not be. But does doing ML on a GMM pose any theoretical issues that could prevent one from finding a solution? Isn't it still possible to find a solution using the EM algorithm?
First regarding the practical issue of vanishing variance most implementations of GMM will have a limit on how small the variance can get, you can set the minimum to some small value based on your specific problem.
Secondly EM is guaranteed to improve on the objective function (Likelihood in the case of ML) or remain the same. After each iteration the parameters are updated. If the parameters have converged you have a local optimum, which is your final solution. The short answer is there is nothing preventing you form finding such a solution - If you have the necessary precautions to prevent variance going to zero.
In such a case (like many others : HMM, ME) you should be careful to use a logistic representation of you likelyhoods, ie store log(v) instead of v for a value v. The important point is to avoid floating point underflow/overflow.