So, getting an "idea" of the optimal number of clusters in k-means is well documented. I found an article on doing this in gaussian mixtures, but not sure I am convinced by it, don't understand it very well. Is there a ... gentler way of doing this?

  • 4
    $\begingroup$ Could you cite the article, or at least outline the methodology it proposes? It's hard to come up with a "gentler" way of doing this if we don't know the baseline :) $\endgroup$ – jbowman Jul 16 '12 at 18:30
  • 1
    $\begingroup$ Geoff McLachlan and others have written books on mixture distributions. I am sure these include approaches to determining the number of components in a mixture. You could probably look there. I agree with jbowman that relieving your confusion would best be achieved if you would indicate to us what it is that you are confused about. $\endgroup$ – Michael R. Chernick Jul 16 '12 at 19:05
  • $\begingroup$ The Estimating Optimal Number of Gaussian Mixtures Based on Incremental k-means for Speaker Identification.... Is its title, it's free to download. It basically increments the number of clusters by 1 until you see that two clusters become dependant between each other, something like that. Thank you! $\endgroup$ – JEquihua Jul 18 '12 at 20:18
  • $\begingroup$ Why not just choose the number of components that maximises the cross-validation estimate of the likelihood? It is computationally expensive, but cross-validation is hard to beat in most cases for model selection, unless there are a large number of parameters to tune. $\endgroup$ – Dikran Marsupial Dec 18 '13 at 15:58
  • $\begingroup$ Can you explain a bit what the cross-validation estimate of the likelihood is? I'm not aware of the concept. Thank you. $\endgroup$ – JEquihua Dec 18 '13 at 19:36

Just some extension to Dikran Marsupial's comment (cross-validation). The main idea is to split your data into training and validation sets in some way, try different number of components and select the best based on the corresponding training and validation likelihood values.

The likelihood for GMM is just $p(x|\pi,\mu,\Sigma)=\sum_K\pi_kN(x|\mu_k,\Sigma_k)$ by definition, where $K$ is the number of components (clusters) and $\pi$,$\mu$,$\Sigma$ are model parameters. By changing the value of $K$ you can plot the GMM likelihood for training and validation sets like the following.

enter image description here

In this example it should be obvious that the optimal number of components is around 20. There's nice video about this on Coursera, and it's where I got the above picture from.

Another commonly used the method is the Bayesian information criterion (BIC): $$BIC = -2\log(L)+K\log(n)$$ where $L$ is the likelihood, K the number of parameters and $n$ the number of data points. It can be understood as adding a penalty for the number of parameters to the log likelihood.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.