Is it important to make a feature scaling before using Gaussian Mixture Model? and why is it important while we are using probability in getting our clusters's parameters (mean and covariance matrix). On the other hand, I know that it's important to standardize our data before using K-mean as clustering here depend on the Ecludiane distance between points and the cluster, and if a feature have large values it would dominate this multidimensional distance calculation


I'm going to assume that you mean , when you say "using a Gaussian Mixture Model", you mean fitting a mixture of (possibly multivariate) Gaussians to some data, for the purposes of clustering.

In this case, provided you use maximum-likelihood as your condition for fitting the model, you don't need to scale your data. If one variable has a higher variance than another, your optimisation procedure will be able to learn this and fit your variances (or covariance matrices in the multivariate case) accordingly.

Only if you include a prior (and are thus doing posterior maximisation) will the scale of your data be important.

The answer why it's important in KMeans and not Gaussian Mixture Models, it's easiest to explain in terms of the soft KMeans algorithm, which KMeans itself is a limiting case of. The soft KMeans algorithm is the same as Gaussian Mixture modelling, if you assume that all of your clusters are generated by Gaussians of the same variance (and no covariance, all features are independent). For that reason, it makes sense to enforce that all your features do have the same variance (but you don't need to centre them, because KMeans allows the distributions to have different centroids, it learns them).

Gaussian Mixture Modelling explicitly relaxes both the assumption of all clusters having the same variance, and the assumption of no correlation of features within a cluster, and that's why you don't need to standardise your features.

To be clear, the real advantage to using Gaussian Mixture Models is that your clusters don't have to be hyper-spherical and of the same radius. The fact that you also don't have to standardise your variables is just a nice bonus


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.