Suppose that we construct a Gaussian Mixture Model with FIXED COVARIANCE on $n$ points using $k=n$ clusters. Is it the case that the Maximum Likelihood parameters put each of the $n$ points in their own clusters, with the cluster means centered at the data points? This seems intuitive, but it is not clear to me that this is necessarily the case. Does anyone have experience with this?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
GMM use maximum likelihood to find the optimal parameter, which depends on latent variable to estimate the cluster id. Its main purpose is to find variance of cluster, which is one of the shortcoming of kmeans. But with same covariance for each cluster or k=n, it become same as the kmean.
And it is sensitive to initialization. If you use good initialization, then it will find good cluster, means each cluster as one different data-point. But with random initialization, it will unable to find good cluster.
With good initilaization:
X = np.random.rand(5,2)
gmm = GaussianMixture(n_components=5, covariance_type='full', init_params='kmeans')
gmm.fit(X)
gmm.means_
[[0.44687696, 0.43307745],
[0.25691664, 0.87355683],
[0.1855179 , 0.53266859],
[0.49189223, 0.89896109],
[0.32626963, 0.31654256]]
With random init:
gmm = GaussianMixture(n_components=5, covariance_type='full', init_params='kmeans')
gmm.fit(X)
gmm.means_
[[0.35258202, 0.59493791],
[0.09373107, 0.51603179],
[0.52300639, 0.32056533],
[0.52300639, 0.32056533],
[0.52300639, 0.32056533]]
As you can see, there are three cluster with one data-point. But with multiple run, you can find result as same as with first one.
-
$\begingroup$ "with the same covariance for each cluster... it becomes the same as kmeans." This is a sufficient condition for the property proposed in the question. I would accept an answer that consists solely of an elaboration on this fact. $\endgroup$– HimCommented Jan 8, 2019 at 14:42
-
1$\begingroup$ sorry for late response, The intuition behind this is as follows, In GMM, our task is to learn mean as well as variance. And for cluster id assignment, we find probability for each data-point for belong to each cluster, where as in kmeans, this happens in hard way only 0 or 1. So with the same variance, we are finding the mean as from the cluster ids, which is nothing but very similar to kmean centroid calculation. Only difference is that in gmm, it is weighted centroid, with the probabilty. But note that mean will be influence by the majority part. For more, you can check Bishop ML book. $\endgroup$ Commented Jan 10, 2019 at 13:49
-
$\begingroup$ "Only difference is that in gmm, it is weighted centroid, with the probabilty." Precisely. This is why it is non obvious that GMM devolves to kmeans when the variances are fixed. I will check the reference you suggest, thank you. $\endgroup$– HimCommented Jan 10, 2019 at 15:00