I am trying to find an automated way of picking the number of clusters $K \in \mathbb{N}$ for unsupervised learning scenarios, specifically for GMM.

I was suggested to use something called the "Bayesian Information Criterion" (BIC) for GMM

The recommended BIC algorithm states,

  1. Let $k = 1$, (start with one cluster)
  2. Run EM algorithm for GMM and obtain mean, variance and mixing coefficients that maximizes the log-likehood, which we refer to as {mean, variance, mixing coefficients}
  3. Plug these values into the "magical" formula

$$BIC(k, N) = -2 \times \text{log-likelihood function} (\text{mean, variance, mixing coefficients}) + \log(N) \times k (2d+1)$$ where $N$ is the number of data points, $d$ is the dimension of the data

  1. $k \leftarrow k + 1$ and repeat (step 1 - 4) until $k$ is very large

  2. Pick the $k$ that gives the minimum BIC value.

I am not sure if this is correct and would like a second pair of eye to verify. In particular, I am not seeing anything that indicates how I should pick my initial values. I am also not sure how long I should run this algorithm for.

Note: a similar formulation for the BIC is given in this question The bayesian information criterion (BIC) Under the Gaussian model

  • $\begingroup$ The last term in the BIC formula is the number of model parameters, which doesn't look right to me in your case - however this depends on your choice of covariance for the mixture model. I laid that out for the case of a full covariance matrix in my answer here. Generally the procedure you describe seems valid though. Also note that you could handle the choice of k by fitting the model via variational inference. $\endgroup$ – deemel Dec 31 '19 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.