# Inference in Dirichlet process mixtures via collapsed Gibbs sampling

I need to cluster some data $$\{x1, \ldots, x_n\}$$ through a Dirichlet process mixture model. Consider the following Dirichlet process mixture model, in which the base measure is a $$NIW(\mu_0, \lambda_0, \nu_0, S)$$ distribution and the random probability measure $$G = \sum_{k=1}^{\infty} \pi_k \delta_{(\mu_k, \Sigma_k)}$$ is constructed according to stick breaking:

$$\pi \sim GEM(\alpha)$$ $$\{\mu_k, \Sigma_k\} \sim NIW(\mu_0, \lambda_0, \nu_0, S)$$ $$z_i | \pi \sim \pi$$ $$x_i | Z_i, \{\mu_k, \Sigma_k\} \sim N(\mu_{z_i}, \Sigma_{z_i})$$

I need to design a collapsed Gibbs sampler. Also, I need to a posteriori estimate $$\{\mu_k, \Sigma_k\}$$ for all clusters in the data. It is clear that this requires integrating out $$\pi$$ and $$\{\mu_k, \Sigma_k\}$$, thus leading to sample only the $$z_i$$'s. However, it is not clear to me how to formally obtain the $$\{\mu_k, \Sigma_k\}$$ subsequently to the collapsed Gibbs sampling process. What about the $$\pi_k$$'s subsequently to the collapsed Gibbs sampling process?

I am not an expert in bayesian non-parametrics but I think I can help a little. Others please correct me If i am wrong.

Based on your model, you would have $$\boldsymbol{\pi}, \mu, \Sigma, \boldsymbol{Z}$$ as your latent variables and $$\boldsymbol{X}$$ as your observables. I use bold symbols to represent vectors. The objective is to find the posterior distribution $$p(\boldsymbol{\pi}, \boldsymbol{Z}, \boldsymbol{\mu}, \boldsymbol{\Sigma}|\boldsymbol{X})$$. Based on the Gibbs sampling procedure, you would have to sample for each latent variable $$\pi_i \sim p(\pi_i|\boldsymbol{\pi_{-i}}, \boldsymbol{\mu},\boldsymbol{\Sigma},\ \boldsymbol{Z},\boldsymbol{X}) \\ (\mu_k, \Sigma_k) \sim p(\mu_k, \Sigma_k|\boldsymbol{\mu_{-k}},\boldsymbol{\Sigma_{-k}}, \boldsymbol{\pi}, \boldsymbol{Z}, \boldsymbol{X})\\ z_i \sim p(z_i | \boldsymbol{Z}_{-i}, \boldsymbol{\Sigma}, \boldsymbol{\mu},\boldsymbol{\pi}, \boldsymbol{X})$$ where the $$-i$$, $$-k$$ subscript denotes the latent variables excluding the $$i$$-th and $$k$$-th variable for that vector. Sampling $$\pi_i$$ is difficult because the Dirichlet Process defines an infinite number of clusters a priori. However, we don't actually need to know the mixture weights $$\pi_i$$ because we are interested in the cluster assignments and the parameters that define each cluster. Hence, we can do a collapsed gibbs sampling procedure by integrating out $$\pi$$ from the complete conditionals. Therefore, we now sample $$(\mu_k, \Sigma_k) \sim p(\mu_k, \Sigma_k|\boldsymbol{\mu_{-k}}, \boldsymbol{\Sigma_{-k}}, \boldsymbol{Z}, \boldsymbol{X})\\ z_i \sim p(z_i|\boldsymbol{Z}_{-i}, \boldsymbol{\mu}, \boldsymbol{\Sigma}, \boldsymbol{X})$$

To sample from these conditional distributions, refer to Radford Neal's paper for a more detailed explanation on how to sample from dirichlet process mixture models.

• You can either use Gibbs sampling (if you are using Conjugate priors) or Metropolis Hastings to sample from these conditional posteriors.

## Gibbs Sampling for DPMM

Suppose that you are using conjugate priors, then when you sample $$z_i$$ there are two outcomes. The first outcome is that you sample a $$z_i = z$$ where $$z$$ is already an existing cluster, the probability of this happening is given by the equation $$P(z_i = z|\boldsymbol{z}_{-i}, x_i, \boldsymbol{\Sigma}, \boldsymbol{\mu}) = b\frac{n_{-i,z}}{n - 1 + \alpha}F(x_i, (\mu_z, \Sigma_z))$$ where $$b$$ is the normalizing constant that makes the conditional distributions sum to 1, $$n_{-i,z}$$ represents the number of points belonging to cluster $$z$$ excluding the $$i$$-th point $$z_i$$, and $$F(x_i, (\mu_z,\Sigma_z))$$ represents the likelihood of the point $$x_i$$ under the NIW distribution with parameters $$\mu_z, \Sigma_z$$.

The second outcome is when you sample a $$z_i = z$$ that is not part of an existing cluster, meaning that the point $$x_i$$ belongs to a new cluster. The probability of sampling a new cluster is given by the equation $$P(z_i = z|\boldsymbol{z}_{-i}, x_i, \boldsymbol{\mu}, \boldsymbol{\Sigma}) = b\frac{\alpha}{n - 1 + \alpha}\int F(x_i, \mu, \Sigma)dG_0(\mu, \Sigma)$$ $$\int F(x_i, \mu, \Sigma)dG_0(\mu, \sigma)$$ is just the prior predictive distribution of the data point $$x_i$$. This integral can be computed exactly because of conjugacy in the likelihood and prior.

With these equations, you can form a complete probability distribution to sample the $$z_i$$'s.

To sample $$(\mu_k, \Sigma_k)$$, you need to sample from the posterior distribution of $$\mu_k, \Sigma_k$$ conditioned on all the data points belonging to cluster $$k$$.

Hope this helps, let me know if you have any questions and please correct me if I am wrong in some points.

• I thank you for your answer. Unfortunately, I still need to see the explicit derivation in the concrete case of my example. Apr 9, 2021 at 12:01
• U can have a look at radford neal's paper for a more detailed derivation of how to integrate out the $\pi$'s, or have a look at this tutorial on DPMM; ncbi.nlm.nih.gov/pmc/articles/PMC6583910 Radford neal's paper should also include an algorithm to that integrates out both $\pi$, $\mu_k, \Sigma_k$. Apr 9, 2021 at 13:14