2
$\begingroup$

Here is a problem that I am looking at.

enter image description here

Is this model really commonly known as a Gaussian mixture model (the one often appears as an illustration of EM algorithm)?

I am confused because Gaussian mixture model (as far as I know) is usually formulated using latent variable, and density function will usually involve some indicator functions. As a specific example, consider the following:

enter image description here

These two models do not look the same to me.

$\endgroup$
  • $\begingroup$ Yes. This is the mixture model. $\endgroup$ – SmallChess Feb 18 '18 at 8:07
  • $\begingroup$ What is the function $\psi$ in your first displayed equation? $\endgroup$ – Dilip Sarwate Feb 18 '18 at 17:04
  • $\begingroup$ @DilipSarwate I'm pretty sure it's a typo made by my professor... $\psi$ is $\phi$ $\endgroup$ – 3x89g2 Feb 18 '18 at 17:06
1
$\begingroup$

The EM algorithm only produces local maxima in the interior of the parameter set. Since the values of the parameters that lead to an infinite likelihood are on the boundary of the parameter set, $\sigma_1=0$ and $\sigma_2=0$, the EM algorithm cannot produce these values unless it is started at these values.

Let me add that, while the likelihood function is not bounded over the parameter set, there exist consistent solutions to the likelihood equations as e.g. the local mode closest to the moment estimator(s).

$\endgroup$
0
$\begingroup$

These problems can indeed occur. They won't happen on nice and clean data, but on real data, this does happen.

Just try a data set with all points on the diagonal (with a full covariance model), or constant attributes, and most EM implementations will fail for exactly this reason.

That is why you may need to use a MAP estimate, where the prior ensures the solution does not degenerate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.