1
$\begingroup$

Is it important to make a feature scaling before using Gaussian Mixture Model? and why is it important while we are using probability in getting our clusters's parameters (mean and covariance matrix). On the other hand, I know that it's important to standardize our data before using K-mean as clustering here depend on the Ecludiane distance between points and the cluster, and if a feature have large values it would dominate this multidimensional distance calculation

$\endgroup$
2
$\begingroup$

I'm going to assume that you mean , when you say "using a Gaussian Mixture Model", you mean fitting a mixture of (possibly multivariate) Gaussians to some data, for the purposes of clustering.

In this case, provided you use maximum-likelihood as your condition for fitting the model, you don't need to scale your data. If one variable has a higher variance than another, your optimisation procedure will be able to learn this and fit your variances (or covariance matrices in the multivariate case) accordingly.

Only if you include a prior (and are thus doing posterior maximisation) will the scale of your data be important.

The answer why it's important in KMeans and not Gaussian Mixture Models, it's easiest to explain in terms of the soft KMeans algorithm, which KMeans itself is a limiting case of. The soft KMeans algorithm is the same as Gaussian Mixture modelling, if you assume that all of your clusters are generated by Gaussians of the same variance (and no covariance, all features are independent). For that reason, it makes sense to enforce that all your features do have the same variance (but you don't need to centre them, because KMeans allows the distributions to have different centroids, it learns them).

Gaussian Mixture Modelling explicitly relaxes both the assumption of all clusters having the same variance, and the assumption of no correlation of features within a cluster, and that's why you don't need to standardise your features.

To be clear, the real advantage to using Gaussian Mixture Models is that your clusters don't have to be hyper-spherical and of the same radius. The fact that you also don't have to standardise your variables is just a nice bonus

$\endgroup$
  • $\begingroup$ Hey @gazza89, nice explanation! Thank you! Could you please provide a reference in the literature on the points you mentioned above? $\endgroup$ – allanvc Jun 16 at 5:07
  • $\begingroup$ I don't have a reference explicitly stating all of the above, but if you want to learn more about the topic, I'd suggest you read the chapter on the EM algorithm in Chris Bishop's book on Pattern Recognition and Machine learning. Additionally to that, why don't you try generating some synthetic data which is a superposition of two bi-variate Gaussians, and then use the EM algorithm assuming a superposition of two bivariate gaussians to learn the parameters. You should see that if you scale the features or do not, the algorithm will learn the corresponding variance parameters. $\endgroup$ – gazza89 Jun 17 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.