Check on intuition behind infinite mixture models for clustering

Question

I'm trying to better understand the intuition and practical application of infinite mixture models (Dirichlet Process) and finite mixture models.

For example, say I have a data set on which I run a Dirichlet Process Mixture Model (using Gibbs/MCMC) and the result is a posterior distribution over number of clusters, and parameter estimates for each cluster. Say the mode of the posterior is $k=10$ clusters. Ideally, in practice, this would be my best guess at the number of clusters.

Now suppose I instead fit a finite mixture model. Let's say I fit $k$ Gaussian Mixture Models (using EM) by looping from $k = 1...n$ (where $n$ is the number of data points). Obviously this is time intensive and inefficient but the point is that I want to find the best $k$. My results may slightly change per run, but imagine I do this many times and find some optimal $k$, on average.

Two questions:

1. Is the $k$ that I should find using both approaches going to be similar? Intuitively it should be since I'm finding some number of Gaussian clusters using each method that fits the data, but one uses Gibbs/MCMC and the other EM. It's not obvious to me that they should be the same since I'm not sure they are "maximizing" the same density per Gibbs/EM iteration.

2. Are the parameters of the resulting $k$ Gaussians in each model going to be roughly equivalent? Again, intuitively each model should find the location of the $k$ cluster centers and their spread.

Is there any reason (mathematical explanation preferred) why these two methods should give very different results?

Answer 1

The two results are not guaranteed to be similar because the two approaches maximize different objective functions. However, they are related and the key difference is in the priors:

In DPM (Bayesian approach), you infer cluster parameters ($\theta := \{\pi,\mu_{1:K},\Sigma_{1:K}\}$) by performing MCMC sampling of $\theta$ according to the posterior distribution $\theta \sim p(\theta \mid X)=p(X \mid \theta)p(\theta)$.

The second approach is a maximum likelihood approach. That is, EM iterations output cluster parameters that maximize the likelihood objective function $\hat \theta = \arg\max_{\theta} p(X\mid\theta)$.

Comparing the two methods, the main difference is that in DPM, you let the priors influence your cluster parameters (i.e., DP for $\pi$, Gaussian for $\mu$, Wishart for $\Sigma$), whereas in the EM approach there is no such prior terms. The prior can directly affect the resulting $K$. The most obvious example is that the hyperparameter, $\alpha$, that you set for DP prior directly controls $K$ (the larger $\alpha$, the bigger output $K$). Keep in mind that, since this is an unsupervised problem, there is no single "correct" $K$ (unless your data have very strong cluster separation and cohesion that the likelihood term simply dominates over the prior term, which can be far from true in real data.). Therefore, in practical, the two results can be quite different, depending on the prior hyperparameters $\alpha$ and $G_0$ that you provide.