Clarifying Dirichlet Process Mixture Probability Terms

Question

Suppose I have a Dirichlet Process Mixture model defined as follows:

$\alpha \sim G(a,b)\\ \pi|\alpha \sim \text{Dir}(\alpha)\\ z|\pi \sim \text{Cat}(\pi)\\ $

where $G$ is just a standard Gamma distribution, and

$ \mu,\Sigma \sim NIW(\beta) $ where $NIW$ stand for Normal Inverse Wishart.

Ultimately the plate model looks like:

Upon doing some probabilistic maniuplation I arrive at the following terms I would like some clarification on: $p(z|X),p(z|X,\alpha),p(X|z)$. I am just wondering how to interpret these.

• Is $p(z|X,\alpha) = p(z|\alpha)$? It seems that $X$ is already implicitly used in the $p(z|\alpha)$ expression as the `number of terms allocated to a cluster' in the sense of the Chinese Restaurant Process, when considering $p(z|\alpha)$ by itself, so does conditioning on $X$ change anything?

• How should I interpret the expressions $p(z|X)$ and $p(X|z)$? It feels strange because $z$ just acts like a `switch' or index for this Bayes Net, so interpreting either direction of these feels strange.

• In addition, if I add a (Gamma) prior distribution over the $\alpha$ term, with a likelihood term similar to that suggested by Escobar & West 1995 (equations 13 / 14) then is it possible to make the claim that this empirical distribution over $\alpha$ has a regularising effect over the amount of clusters forming (or in the cluster allocations for each point, that is $p(z_i = k | X, \beta, \alpha, z_{\lnot i}$).

Answer 1

Let's review the generative process assumed to generate data using GMM with infinite (i.e. non-fixed) number of clusters.

1. First we need to choose a cluster assignment. Using a Chinese restaurant process, we assume that there are currently $K$ clusters, but we also have a probability to assign an observation to a new cluster $K+1$. This way we do not need to fix $K$ a priori,

$$ \begin{equation}\tag{1}\label{eqn1} P(z_i=k\mid \alpha) = \begin{cases} \frac{N_k}{N+\alpha-1} & , k \in [1, K]\\ \frac{\alpha}{N+\alpha-1} & , k = k_{new}=K+1\\ \end{cases} \end{equation} $$

2. Given a cluster assignment, we can generate an observation from the corresponding Gaussian with parameters: $\mathcal{N}(\mu_k, \Sigma_k)$

For clustering, we need to determine the probability of assigning an observation to a cluster, which can be done using Bayes rule as follows,

$$\tag{2}\label{eqn2} P(\textbf{z} \mid \textbf{X}, \alpha, \beta) \propto P(\textbf{X} \mid \textbf{z}, \beta) \times P(\textbf{z} \mid \alpha) $$

If we are doing MCMC sampling, the above equation can be written as,

$$\tag{3}\label{eqn3} P(z_i=k \mid \textbf{z}_{-i}, \textbf{X}, \alpha, \beta) \propto P(x_i \mid \textbf{X}_{-i}, z_i=k, \textbf{z}_{-i}, \beta) \times P(z_i=k \mid \textbf{z}_{-i}, \alpha) $$

---

Answering your questions:

Is $P(\textbf{z} \mid \textbf{X}, \alpha, \beta) = P(\textbf{z} \mid \alpha)$?

No they are not. $P(\textbf{z} \mid \textbf{X}, \alpha, \beta)$ is the posterior evaluated using equation $(\ref{eqn2})$ and you get $P(\textbf{z} \mid \alpha)$ by integrating out $\pi$ as follows:

$$ P(\textbf{z} \mid \alpha) = \int_{\pi}P(\textbf{z} \mid \mathbb{\pi}) \: p(\mathbb{\pi} \mid \alpha) \: d\pi $$

How should I interpret the expressions $P(\textbf{z} \mid \textbf{X})$ and $P(\textbf{X} \mid \textbf{z})$?

The former term is the posterior distribution $[$ i.e. which is (\ref{eqn2}) ignoring $\alpha$ and $\beta$ $]$ of assigning clusters to the data, the latter is the likelihood of the data. The posterior is calculated using the two terms:

• $P(z_i=k \mid \alpha)$ is given by equation $(\ref{eqn1})$
• $P(x_i \mid z_i=k, \beta) \sim \mathcal{N}(\mu_k, \Sigma_k)$

... then is it possible to make the claim that this empirical distribution over $\alpha$ has a regularising effect over the amount of clusters forming ...

Your analysis is correct, since the choice of $\alpha$ governs how many clusters will be allocated and whether the model will have a tendency to favor existing clusters or generate more clusters.

---

References:

1. Gibbs sampling for fitting finite and infinite Gaussian mixture models - by Herman Kamper

2. A tutorial on Bayesian nonparametric models - by Gershman, Blei