Let's say I have some [multivariate] data and want to fit a GMM to it. So I have $P_x=\sum_{i=1}^{n}\alpha_i{N(x;\theta_i)}$, where $x$ is an observation from the data, $\theta_i$ is the mean and covariance matrix parameters for the ith Gaussian, and $\sum_{i=1}^{n}\alpha_i=1$, i.e. it is a contrast to ensure a valid probability distribution of the mixed Gaussians.

As we know this would be easy to setup a liklihood for and optimize via maximum liklihood. The optimized result would be a local optimum and a valid probability distribution (I could ensure a global optimum by doing something like self-contrastive estimation as described by Ian Goodfellow), but admittedly now I'm a bit stuck on interpretation.

The mixing weights $\alpha_i$ seem like they would represent the marginal probability of each group (i.e., $\alpha_i$ would be the marginal probability of group "i"...$P(group_i)$), but then the ith Gaussian would be like the likelihood of group "i" given the data. I.e., $P(group_1|x), P(group_2|x)$, etc. which when summed would be like the normalizing constant for $P(x|group_1), P(x|group_2)$, etc... Or is the output of the ith Gaussian $P(x|group_i)$ (which would make more sense...)? If the latter, then since $P(group_i|x)=\frac{P(x|group_i)P(group_i)}{P(x)}$ is the output for EM algorithm model I could seemingly back calculate very easily what EM algorithm would provide.

Any ideas if I'm viewing this correctly. It seems if I just want to fit a flexible PDF to my data my method would work though, correct? Still trying to reconcile the difference when applying EM algorithm though to the same type of problem, even if the method I describe is legit...

  • 1
    $\begingroup$ The EM algorithm is maximum likelihood. The advantage is that one can use the method-of-moments estimators to maximize the likelihood conditional on the group assignment. I suppose one could write out the very irregular joint likelihood of the group assignments and means and do some kind of grid search, but this would be unpractical due to computational complexity. $\endgroup$
    – AdamO
    Feb 1, 2018 at 21:20
  • $\begingroup$ Yes but it seems like EM has some kind of extra step of finding which group each data point belongs to on each iteration. What I'm describing doesn't entail that...which is one reason I'm getting a bit lost. There's no explicit need to assign a group in what I've detailed. $\endgroup$
    – JPJ
    Feb 1, 2018 at 21:22
  • 2
    $\begingroup$ Frankly I don't understand what you're proposing, nor can I tell if it's actually different from EM. Why don't you set up an implementation? $\endgroup$
    – AdamO
    Feb 1, 2018 at 21:29
  • $\begingroup$ LOL....noted. I'm essentially proposing a neural network with Gaussian activation functions but stipulating that the activation weights will be structured/designed to be the covariances and means, and the output weights would form a contrast to ensure a valid probability distribution. I WILL setup an implementation because my main goal is density estimation, but asking this in parallel since interpret ability and connecting to existing methods is always very nice :-) $\endgroup$
    – JPJ
    Feb 1, 2018 at 21:36
  • 3
    $\begingroup$ Possible duplicate of Why is optimizing a mixture of Gaussian directly computationally hard? $\endgroup$
    – Xi'an
    Feb 2, 2018 at 7:25

1 Answer 1


After some searching around pretty hard I found a solid answer to my question of why not just use plain old ML instead of EM algorithm in a book called Pattern Recognition and ML by Christopher Bishop.

Excerpt copied exactly below from pages 433,434 (hopefully this is OK to do since I referenced/gave credit to the source). Note he has a figure reference that helps.

Before discussing how to maximize [a GMM likelihood] function, it is worth emphasizing that there is a significant problem associated with the maximum likelihood framework applied to Gaussian mixture models, due to the presence of singularities. For simplicity, consider a Gaussian mixture whose components have covariance matrices given by $\sum{k}=\sigma^2I$, where $I$ is the unit matrix, although the conclusions will hold for general covariance matrices. Suppose that one of the components of the mixture model, let us say the j-th component, has its mean $u_j$ exactly equal to one of the data points so that $µ_j = x_n$ for some value of n. This data point will then contribute a term in the likelihood function of the form $N(x_n|x_n, σ^2_j I) = \frac{1}{(2π)^{1/2}}\frac{1}{σ_j}$. If we consider the limit $σ_j → 0$, then we see that this term goes to infinity and so the log likelihood function will also go to infinity. Thus the maximization of the log likelihood function is not a well posed problem because such singularities will always be present and will occur whenever one of the Gaussian components ‘collapses’ onto a specific data point. Recall that this problem did not arise in the case of a single Gaussian distribution. To understand the difference, note that if a single Gaussian collapses onto a data point it will contribute multiplicative factors to the likelihood function arising from the other data points and these factors will go to zero exponentially fast, giving an overall likelihood that goes to zero rather than infinity. However, once we have (at least) two components in the mixture, one of the components can have a finite variance and therefore assign finite probability to all of the data points while the other component can shrink onto one specific data point and thereby contribute an ever increasing additive value to the log likelihood. This is illustrated in Figure 9.7. These singularities provide another example of the severe over-fitting that can occur in a maximum likelihood approach. We shall see Section 10.1 that this difficulty does not occur if we adopt a Bayesian approach. For the moment, however, we simply note that in applying maximum likelihood to Gaussian mixture models we must take steps to avoid finding such pathological solutions and instead seek local maxima of the likelihood function that are well behaved.

  • 4
    $\begingroup$ The quote is enlightening in general, but it doesn't address the question because the EM algorithm also tries to maximize the the likelihood; it's not distinct from "plain old" maximum likelihood in this regard. All likelihood maximizing algorithms are susceptible to the singularities that Bishop described, including EM. If you were to initialize it in this regime, it wouldn't escape the pathological solution. Conversely, other optimization algorithms that seek a local maximum can find the same good solutions as EM when all are given a reasonable initialization. $\endgroup$
    – user20160
    Mar 6, 2022 at 19:41
  • $\begingroup$ Yes the question is probably more about EM vs Gradient Decent optimization, but nonetheless the above is a useful intuition of one of the problems encountered. See related answers here: Q1 and Q2 $\endgroup$
    – JPJ
    Mar 6, 2022 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.