I am studying the Gaussian Mixture model and come up with this question myself.

Suppose the underlying data is generated from a mixture of $K$ Gaussian distribution and each of them has a mean vector $\mu_k\in\mathbb{R}^p$, where $1\leq k\leq K$ and each of them has the same co-variance matrix $\Sigma$ and assume this $\Sigma$ is a diagonal matrix. And assume the mixing ratio is $1/K$, i.e., each cluster has same weight.

So in this ideal example, the only job is to estimate the $K$ mean vectors $\mu_k\in\mathbb{R}^p$, where $1\leq k\leq K$ and the co-variance matrix $\Sigma$.

My question is: if we use EM algorithm, will we be able to consistently estimate $\mu_k$ and $\Sigma$, i.e., when sample size $n\rightarrow\infty$, will the estimator produced by EM algorithm achieve the true value of $\mu_k$ and $\Sigma$?


If the algorithm is initialized with random values each time, then no, the convergence will not necessarily be consistent. Non-random initialization will presumably produce the same result every time, but I don't believe that this would necessary produce the "correct" values of $\mu_k$.

As an aside, by fixing the mixing ratio to $1/K$ and fixing $\Sigma$ to be diagonal, the algorithm becomes very similar to the $k$-means algorithm. This also has inconsistent convergence, depending on the random initialization.

  • $\begingroup$ I numerically experimented, at least for 2 independent classes of normal distribution, the EM produces consistent estimator of the class mean. However, K means cannot do that, I proved it mathematically $\endgroup$ – KevinKim Sep 13 '15 at 2:39
  • 1
    $\begingroup$ Could you give more details please? E.g. what data you were using, how you initialised the parameters etc. $\endgroup$ – dcorney Sep 14 '15 at 8:58
  • $\begingroup$ Agree with @dcorney. It really depends on the initial values you will choose. At least in practice wrong choice of initial values lead to unconsistent estimation (I use mixtools R package) $\endgroup$ – German Demidov Sep 8 '16 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.