10
$\begingroup$

I am studying the Gaussian Mixture model and come up with this question myself.

Suppose the underlying data is generated from a mixture of $K$ Gaussian distribution and each of them has a mean vector $\mu_k\in\mathbb{R}^p$, where $1\leq k\leq K$ and each of them has the same co-variance matrix $\Sigma$ and assume this $\Sigma$ is a diagonal matrix. And assume the mixing ratio is $1/K$, i.e., each cluster has same weight.

So in this ideal example, the only job is to estimate the $K$ mean vectors $\mu_k\in\mathbb{R}^p$, where $1\leq k\leq K$ and the co-variance matrix $\Sigma$.

My question is: if we use EM algorithm, will we be able to consistently estimate $\mu_k$ and $\Sigma$, i.e., when sample size $n\rightarrow\infty$, will the estimator produced by EM algorithm achieve the true value of $\mu_k$ and $\Sigma$?

$\endgroup$

1 Answer 1

1
$\begingroup$

If the algorithm is initialized with random values each time, then no, the convergence will not necessarily be consistent. Non-random initialization will presumably produce the same result every time, but I don't believe that this would necessary produce the "correct" values of $\mu_k$.

As an aside, by fixing the mixing ratio to $1/K$ and fixing $\Sigma$ to be diagonal, the algorithm becomes very similar to the $k$-means algorithm. This also has inconsistent convergence, depending on the random initialization.

$\endgroup$
3
  • $\begingroup$ I numerically experimented, at least for 2 independent classes of normal distribution, the EM produces consistent estimator of the class mean. However, K means cannot do that, I proved it mathematically $\endgroup$
    – KevinKim
    Sep 13, 2015 at 2:39
  • 1
    $\begingroup$ Could you give more details please? E.g. what data you were using, how you initialised the parameters etc. $\endgroup$
    – dcorney
    Sep 14, 2015 at 8:58
  • $\begingroup$ Agree with @dcorney. It really depends on the initial values you will choose. At least in practice wrong choice of initial values lead to unconsistent estimation (I use mixtools R package) $\endgroup$ Sep 8, 2016 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.