1
$\begingroup$

I am currently reading about the guassian mixture model and the expectation–maximization algorithm. From what I am reading the two differences between the two here is what I've come up with so far, are they correct?

  • GMM is a model, it's a way to represent the data (subpopulations of the entire population)
  • EM is an algorithm, it is a process to represent a specific model (GMM in this case)

My final question is, let's say you have a population, I've been reading that you must define k (amount of clusters) before using an algorithm (EM in this case). Is this related to supervised vs. unsupervised algorithms? In other words, in supervised algorithms you define how many clusters there are in advance and train the cluster, while unsupervised would calculate the amount of clusters it's own. Is this the reason why you must define k?

$\endgroup$
1
  • $\begingroup$ Supervised v Unsupervised has to do with whether the algorithm uses training data that is labeled or not, respectively. You're correct that specifying or learning K has a supervised or unsupervised flavor to it, but that application of the terms would be non-standard and misleading. $\endgroup$
    – Wayne
    Dec 2, 2014 at 16:00

1 Answer 1

1
$\begingroup$

First, you're right that GMM is a model. EM is not a model. EM helps to find values of variables within a model.

That you have to define $k$ before using EM is not itself to do with supervised/unsupervisedness. AFAIK it is to do with the fact that the standard EM algorithm supposes you know the structure of the model / what parameters are involved.

The finite-GMM is a model for the structure of a dataset. EM helps to fit the finite-GMM to a particular dataset. Once you have the structure you can use that as a springboard for doing a classification/regression task.

By way of contrast, you can have non-parametric clustering algorithms for regression. I.e. where you do not specify $k$, and both the number of clusters and cluster assignments are learnt from the data. Other algorithms are used for inference in such models e.g. MCMC.

Supervised/unsupervised is about whether you're trying to minimize some kind of loss function by the settings in a model, or whether you're just learn about the structure of the data itself.

$\endgroup$
2
  • $\begingroup$ Ah ok. So you can't just throw any algorithm at a data set and expect the algorithm to work. You must guess the model first to be able to choose the algorithm depending on what outcome you want (EM for example will help define subpopulations in the large dataset). $\endgroup$ Dec 2, 2014 at 16:04
  • $\begingroup$ More like 'you must guess $k$ if you want to use EM to fit the mixture'. Where guess might involve something like getting the AIC for a bunch of different $k$, then going with the best one. If you don't want to guess $k$ one way of proceeding is to do a Bayesian Dirichlet Process mixture model with fitting/inference done via MCMC. $\endgroup$ Dec 2, 2014 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.