So, getting an "idea" of the optimal number of clusters in k-means is well documented. I found an article on doing this in gaussian mixtures, but not sure I am convinced by it, don't understand it very well. Is there a ... gentler way of doing this?
$\begingroup$
$\endgroup$
5
-
4$\begingroup$ Could you cite the article, or at least outline the methodology it proposes? It's hard to come up with a "gentler" way of doing this if we don't know the baseline :) $\endgroup$– jbowmanJul 16, 2012 at 18:30
-
1$\begingroup$ Geoff McLachlan and others have written books on mixture distributions. I am sure these include approaches to determining the number of components in a mixture. You could probably look there. I agree with jbowman that relieving your confusion would best be achieved if you would indicate to us what it is that you are confused about. $\endgroup$– Michael R. ChernickJul 16, 2012 at 19:05
-
$\begingroup$ The Estimating Optimal Number of Gaussian Mixtures Based on Incremental k-means for Speaker Identification.... Is its title, it's free to download. It basically increments the number of clusters by 1 until you see that two clusters become dependant between each other, something like that. Thank you! $\endgroup$– JEquihuaJul 18, 2012 at 20:18
-
$\begingroup$ Why not just choose the number of components that maximises the cross-validation estimate of the likelihood? It is computationally expensive, but cross-validation is hard to beat in most cases for model selection, unless there are a large number of parameters to tune. $\endgroup$– Dikran MarsupialDec 18, 2013 at 15:58
-
$\begingroup$ Can you explain a bit what the cross-validation estimate of the likelihood is? I'm not aware of the concept. Thank you. $\endgroup$– JEquihuaDec 18, 2013 at 19:36
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Just some extension to Dikran Marsupial's comment (cross-validation). The main idea is to split your data into training and validation sets in some way, try different number of components and select the best based on the corresponding training and validation likelihood values.
The likelihood for GMM is just $p(x|\pi,\mu,\Sigma)=\sum_K\pi_kN(x|\mu_k,\Sigma_k)$ by definition, where $K$ is the number of components (clusters) and $\pi$,$\mu$,$\Sigma$ are model parameters. By changing the value of $K$ you can plot the GMM likelihood for training and validation sets like the following.
In this example it should be obvious that the optimal number of components is around 20. There's nice video about this on Coursera, and it's where I got the above picture from.
Another commonly used the method is the Bayesian information criterion (BIC): $$BIC = -2\log(L)+K\log(n)$$ where $L$ is the likelihood, K the number of parameters and $n$ the number of data points. It can be understood as adding a penalty for the number of parameters to the log likelihood.
