Suppose that we construct a Gaussian Mixture Model with FIXED COVARIANCE on $n$ points using $k=n$ clusters. Is it the case that the Maximum Likelihood parameters put each of the $n$ points in their own clusters, with the cluster means centered at the data points? This seems intuitive, but it is not clear to me that this is necessarily the case. Does anyone have experience with this?
GMM use maximum likelihood to find the optimal parameter, which depends on latent variable to estimate the cluster id. Its main purpose is to find variance of cluster, which is one of the shortcoming of kmeans. But with same covariance for each cluster or k=n, it become same as the kmean.
And it is sensitive to initialization. If you use good initialization, then it will find good cluster, means each cluster as one different data-point. But with random initialization, it will unable to find good cluster.
With good initilaization:
X = np.random.rand(5,2) gmm = GaussianMixture(n_components=5, covariance_type='full', init_params='kmeans') gmm.fit(X) gmm.means_ [[0.44687696, 0.43307745], [0.25691664, 0.87355683], [0.1855179 , 0.53266859], [0.49189223, 0.89896109], [0.32626963, 0.31654256]]
With random init:
gmm = GaussianMixture(n_components=5, covariance_type='full', init_params='kmeans') gmm.fit(X) gmm.means_ [[0.35258202, 0.59493791], [0.09373107, 0.51603179], [0.52300639, 0.32056533], [0.52300639, 0.32056533], [0.52300639, 0.32056533]]
As you can see, there are three cluster with one data-point. But with multiple run, you can find result as same as with first one.