2
$\begingroup$

My question concerns the expectation-maximisation algorithm used to estimate the hyper-parameters of a Gaussian mixture model in z multivariate setup. I understand that the EM algorithm uses the Maximum likelihood criterion as starting point, but I was wondering about the metric used (implicitly) in such technique.

For different clustering techniques like "Hierarchical clustering" or "k-means", a distance is used (either Euclidian or a different one) to assess the "closeness" of two points; I get that learning a GMM aims at maximizing the likelihood between the model and the training data, but when looked at as an unsupervised clustering technique, is it possible to associate a "hidden" metric to the EM procedure ? And if the answer is yes, could it be changed (maximizing likelihood while taking into account a certain notion of neigborhood between features) ?

Thanks

$\endgroup$
3
$\begingroup$

Just some extension to Anony-Mousse's answer.

K-means is a particular case of EM for GMM that assumes
1) a hard assignment of data points to clusters (the hidden variables), and
2) that the identity matrix is used as the covariance.

Multivariate Gaussian with identity covariance matrix becomes $$\frac{1}{(2\pi\epsilon)^{1/2}}exp\{-\frac{1}{2\epsilon}\lVert x-\mu\rVert^2 \},$$ so minimizing the log likelihood is equivalent to minimizing the squared Euclidean distance.

If we use an arbitrary covariance matrix, the exponential term of Gaussian $$-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)$$ can be seen as the squared Mahalanobis distance.

The distribution will be no longer Gaussian if the metric is changed.

$\endgroup$
2
$\begingroup$

The "hidden" metric is not at all hidden. It is the Mahalanobis distance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.