I need to cluster some data $\{x1, \ldots, x_n\}$ through a Dirichlet process mixture model. Consider the following Dirichlet process mixture model, in which the base measure is a $NIW(\mu_0, \lambda_0, \nu_0, S)$ distribution and the random probability measure $G = \sum_{k=1}^{\infty} \pi_k \delta_{(\mu_k, \Sigma_k)}$ is constructed according to stick breaking:

$$\pi \sim GEM(\alpha)$$ $$ \{\mu_k, \Sigma_k\} \sim NIW(\mu_0, \lambda_0, \nu_0, S) $$ $$ z_i | \pi \sim \pi$$ $$x_i | Z_i, \{\mu_k, \Sigma_k\} \sim N(\mu_{z_i}, \Sigma_{z_i})$$

I need to design a collapsed Gibbs sampler. Also, I need to a posteriori estimate $\{\mu_k, \Sigma_k\}$ for all clusters in the data. It is clear that this requires integrating out $\pi$ and $ \{\mu_k, \Sigma_k\}$, thus leading to sample only the $z_i$'s. However, it is not clear to me how to formally obtain the $\{\mu_k, \Sigma_k\}$ subsequently to the collapsed Gibbs sampling process. What about the $\pi_k$'s subsequently to the collapsed Gibbs sampling process?


1 Answer 1


I am not an expert in bayesian non-parametrics but I think I can help a little. Others please correct me If i am wrong.

Based on your model, you would have $\boldsymbol{\pi}, \mu, \Sigma, \boldsymbol{Z}$ as your latent variables and $\boldsymbol{X}$ as your observables. I use bold symbols to represent vectors. The objective is to find the posterior distribution $p(\boldsymbol{\pi}, \boldsymbol{Z}, \boldsymbol{\mu}, \boldsymbol{\Sigma}|\boldsymbol{X})$. Based on the Gibbs sampling procedure, you would have to sample for each latent variable $$\pi_i \sim p(\pi_i|\boldsymbol{\pi_{-i}}, \boldsymbol{\mu},\boldsymbol{\Sigma},\ \boldsymbol{Z},\boldsymbol{X}) \\ (\mu_k, \Sigma_k) \sim p(\mu_k, \Sigma_k|\boldsymbol{\mu_{-k}},\boldsymbol{\Sigma_{-k}}, \boldsymbol{\pi}, \boldsymbol{Z}, \boldsymbol{X})\\ z_i \sim p(z_i | \boldsymbol{Z}_{-i}, \boldsymbol{\Sigma}, \boldsymbol{\mu},\boldsymbol{\pi}, \boldsymbol{X}) $$ where the $-i$, $-k$ subscript denotes the latent variables excluding the $i$-th and $k$-th variable for that vector. Sampling $\pi_i$ is difficult because the Dirichlet Process defines an infinite number of clusters a priori. However, we don't actually need to know the mixture weights $\pi_i$ because we are interested in the cluster assignments and the parameters that define each cluster. Hence, we can do a collapsed gibbs sampling procedure by integrating out $\pi$ from the complete conditionals. Therefore, we now sample $$ (\mu_k, \Sigma_k) \sim p(\mu_k, \Sigma_k|\boldsymbol{\mu_{-k}}, \boldsymbol{\Sigma_{-k}}, \boldsymbol{Z}, \boldsymbol{X})\\ z_i \sim p(z_i|\boldsymbol{Z}_{-i}, \boldsymbol{\mu}, \boldsymbol{\Sigma}, \boldsymbol{X}) $$

To sample from these conditional distributions, refer to Radford Neal's paper for a more detailed explanation on how to sample from dirichlet process mixture models.

  • You can either use Gibbs sampling (if you are using Conjugate priors) or Metropolis Hastings to sample from these conditional posteriors.

Gibbs Sampling for DPMM

Suppose that you are using conjugate priors, then when you sample $z_i$ there are two outcomes. The first outcome is that you sample a $z_i = z$ where $z$ is already an existing cluster, the probability of this happening is given by the equation $$ P(z_i = z|\boldsymbol{z}_{-i}, x_i, \boldsymbol{\Sigma}, \boldsymbol{\mu}) = b\frac{n_{-i,z}}{n - 1 + \alpha}F(x_i, (\mu_z, \Sigma_z)) $$ where $b$ is the normalizing constant that makes the conditional distributions sum to 1, $n_{-i,z}$ represents the number of points belonging to cluster $z$ excluding the $i$-th point $z_i$, and $F(x_i, (\mu_z,\Sigma_z))$ represents the likelihood of the point $x_i$ under the NIW distribution with parameters $\mu_z, \Sigma_z$.

The second outcome is when you sample a $z_i = z$ that is not part of an existing cluster, meaning that the point $x_i$ belongs to a new cluster. The probability of sampling a new cluster is given by the equation $$P(z_i = z|\boldsymbol{z}_{-i}, x_i, \boldsymbol{\mu}, \boldsymbol{\Sigma}) = b\frac{\alpha}{n - 1 + \alpha}\int F(x_i, \mu, \Sigma)dG_0(\mu, \Sigma) $$ $\int F(x_i, \mu, \Sigma)dG_0(\mu, \sigma)$ is just the prior predictive distribution of the data point $x_i$. This integral can be computed exactly because of conjugacy in the likelihood and prior.

With these equations, you can form a complete probability distribution to sample the $z_i$'s.

To sample $(\mu_k, \Sigma_k)$, you need to sample from the posterior distribution of $\mu_k, \Sigma_k$ conditioned on all the data points belonging to cluster $k$.

Hope this helps, let me know if you have any questions and please correct me if I am wrong in some points.

  • $\begingroup$ I thank you for your answer. Unfortunately, I still need to see the explicit derivation in the concrete case of my example. $\endgroup$ Apr 9, 2021 at 12:01
  • $\begingroup$ U can have a look at radford neal's paper for a more detailed derivation of how to integrate out the $\pi$'s, or have a look at this tutorial on DPMM; ncbi.nlm.nih.gov/pmc/articles/PMC6583910 Radford neal's paper should also include an algorithm to that integrates out both $\pi$, $\mu_k, \Sigma_k$. $\endgroup$
    – calveeen
    Apr 9, 2021 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.