Is it important to make a feature scaling before using Gaussian Mixture Model? and why is it important while we are using probability in getting our clusters's parameters (mean and covariance matrix). On the other hand, I know that it's important to standardize our data before using K-mean as clustering here depend on the Ecludiane distance between points and the cluster, and if a feature have large values it would dominate this multidimensional distance calculation
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
I'm going to assume that you mean , when you say "using a Gaussian Mixture Model", you mean fitting a mixture of (possibly multivariate) Gaussians to some data, for the purposes of clustering.
In this case, provided you use maximum-likelihood as your condition for fitting the model, you don't need to scale your data. If one variable has a higher variance than another, your optimisation procedure will be able to learn this and fit your variances (or covariance matrices in the multivariate case) accordingly.
Only if you include a prior (and are thus doing posterior maximisation) will the scale of your data be important.
To answer why it's important in KMeans and not Gaussian Mixture Models, it's easiest to explain in terms of the soft KMeans algorithm, which KMeans itself is a limiting case of. The soft KMeans algorithm is the same as Gaussian Mixture modelling, if you assume that all of your clusters are generated by Gaussians of the same variance (and no covariance, all features are independent). For that reason, it makes sense to enforce that all your features do have the same variance (but you don't need to centre them, because KMeans allows the distributions to have different centroids, it learns them).
Gaussian Mixture Modelling explicitly relaxes both the assumption of all clusters having the same variance, and the assumption of no correlation of features within a cluster, and that's why you don't need to standardise your features.
To be clear, the real advantage to using Gaussian Mixture Models is that your clusters don't have to be hyper-spherical and of the same radius. The fact that you also don't have to standardise your variables is just a nice bonus
-
$\begingroup$ Hey @gazza89, nice explanation! Thank you! Could you please provide a reference in the literature on the points you mentioned above? $\endgroup$– allanvcCommented Jun 16, 2019 at 5:07
-
$\begingroup$ I don't have a reference explicitly stating all of the above, but if you want to learn more about the topic, I'd suggest you read the chapter on the EM algorithm in Chris Bishop's book on Pattern Recognition and Machine learning. Additionally to that, why don't you try generating some synthetic data which is a superposition of two bi-variate Gaussians, and then use the EM algorithm assuming a superposition of two bivariate gaussians to learn the parameters. You should see that if you scale the features or do not, the algorithm will learn the corresponding variance parameters. $\endgroup$– gazza89Commented Jun 17, 2019 at 9:02
-
$\begingroup$ is this also valid if you use k-means for initalization of the clusters instead of for example random initalization? $\endgroup$– AmonetCommented Jun 29, 2020 at 16:14
-
2$\begingroup$ if you use KMeans to initialise, you will get a different initialisation for scaled and non-scaled features, so the answer is "it will definitely make a difference". One would hope that the EM algorithm for GMMs isn't super sensitive to the initialisation so it wouldn't matter too much for the final answer, but this isn't guaranteed. $\endgroup$– gazza89Commented Jun 30, 2020 at 10:32
-
$\begingroup$ Thanks, I trained a GMM multiple times using the same training and test set and ended up with idential results on the test set. So I guess, as you say, it's not too sensitive (at least given my data). $\endgroup$– AmonetCommented Jul 13, 2020 at 9:16