# Clarifying Dirichlet Process Mixture Probability Terms

Suppose I have a Dirichlet Process Mixture model defined as follows:

$$\alpha \sim G(a,b)\\ \pi|\alpha \sim \text{Dir}(\alpha)\\ z|\pi \sim \text{Cat}(\pi)\\$$

where $$G$$ is just a standard Gamma distribution, and

$$\mu,\Sigma \sim NIW(\beta)$$ where $$NIW$$ stand for Normal Inverse Wishart.

Ultimately the plate model looks like: Upon doing some probabilistic maniuplation I arrive at the following terms I would like some clarification on: $$p(z|X),p(z|X,\alpha),p(X|z)$$. I am just wondering how to interpret these.

• Is $$p(z|X,\alpha) = p(z|\alpha)$$? It seems that $$X$$ is already implicitly used in the $$p(z|\alpha)$$ expression as the number of terms allocated to a cluster' in the sense of the Chinese Restaurant Process, when considering $$p(z|\alpha)$$ by itself, so does conditioning on $$X$$ change anything?

• How should I interpret the expressions $$p(z|X)$$ and $$p(X|z)$$? It feels strange because $$z$$ just acts like a switch' or index for this Bayes Net, so interpreting either direction of these feels strange.

• In addition, if I add a (Gamma) prior distribution over the $$\alpha$$ term, with a likelihood term similar to that suggested by Escobar & West 1995 (equations 13 / 14) then is it possible to make the claim that this empirical distribution over $$\alpha$$ has a regularising effect over the amount of clusters forming (or in the cluster allocations for each point, that is $$p(z_i = k | X, \beta, \alpha, z_{\lnot i}$$).

• Why do you have an arrow between $\mu_K$ and $\Sigma_K$ nodes? – kedarps Apr 23 at 6:15
• This is common in normal inverse wishart models where the output of $\Sigma_k$ can model the variance over the $\mu_k$, see the wiki en.wikipedia.org/wiki/Normal-inverse-Wishart_distribution – pche8701 Apr 23 at 6:17
• Yes, I am aware of that. But you draw both $\mu_K$ and $\Sigma_K$ from NIW. With the arrow you are implying that a draw of $\mu_K$ depends on the draw of $\Sigma_K$ -- which is not true. Hence there is no need for an arrow between the two nodes. – kedarps Apr 23 at 6:24
• I am probably wrong, but I just always assumed that the arrows point in the direction of dependency, and $\mu_k$ does depend on $\Sigma_k$. Is this an incorrect understanding? – pche8701 Apr 23 at 6:34
• This is a minor point and not so important from the context of your question in the post. You are correct, that if you are drawing $\mu_K$ and $\Sigma_K$ separately, you would draw $\Sigma_K$ from Inverse-Wishart distribution followed by $\mu_K$ from a Normal distribution. Since you are specifying that $(\mu_K, \Sigma_K)$ are drawn from $NIW(\beta)$, you can just skip the arrow between the two to simplify the notation. – kedarps Apr 23 at 6:45

## 1 Answer

Let's review the generative process assumed to generate data using GMM with infinite (i.e. non-fixed) number of clusters.

1. First we need to choose a cluster assignment. Using a Chinese restaurant process, we assume that there are currently $$K$$ clusters, but we also have a probability to assign an observation to a new cluster $$K+1$$. This way we do not need to fix $$K$$ a priori,

$$\begin{equation}\tag{1}\label{eqn1} P(z_i=k\mid \alpha) = \begin{cases} \frac{N_k}{N+\alpha-1} & , k \in [1, K]\\ \frac{\alpha}{N+\alpha-1} & , k = k_{new}=K+1\\ \end{cases} \end{equation}$$

1. Given a cluster assignment, we can generate an observation from the corresponding Gaussian with parameters: $$\mathcal{N}(\mu_k, \Sigma_k)$$

For clustering, we need to determine the probability of assigning an observation to a cluster, which can be done using Bayes rule as follows,

$$\tag{2}\label{eqn2} P(\textbf{z} \mid \textbf{X}, \alpha, \beta) \propto P(\textbf{X} \mid \textbf{z}, \beta) \times P(\textbf{z} \mid \alpha)$$

If we are doing MCMC sampling, the above equation can be written as,

$$\tag{3}\label{eqn3} P(z_i=k \mid \textbf{z}_{-i}, \textbf{X}, \alpha, \beta) \propto P(x_i \mid \textbf{X}_{-i}, z_i=k, \textbf{z}_{-i}, \beta) \times P(z_i=k \mid \textbf{z}_{-i}, \alpha)$$

## Answering your questions:

Is $$P(\textbf{z} \mid \textbf{X}, \alpha, \beta) = P(\textbf{z} \mid \alpha)$$?

No they are not. $$P(\textbf{z} \mid \textbf{X}, \alpha, \beta)$$ is the posterior evaluated using equation $$(\ref{eqn2})$$ and you get $$P(\textbf{z} \mid \alpha)$$ by integrating out $$\pi$$ as follows:

$$P(\textbf{z} \mid \alpha) = \int_{\pi}P(\textbf{z} \mid \mathbb{\pi}) \: p(\mathbb{\pi} \mid \alpha) \: d\pi$$

How should I interpret the expressions $$P(\textbf{z} \mid \textbf{X})$$ and $$P(\textbf{X} \mid \textbf{z})$$?

The former term is the posterior distribution $$[$$ i.e. which is (\ref{eqn2}) ignoring $$\alpha$$ and $$\beta$$ $$]$$ of assigning clusters to the data, the latter is the likelihood of the data. The posterior is calculated using the two terms:

• $$P(z_i=k \mid \alpha)$$ is given by equation $$(\ref{eqn1})$$
• $$P(x_i \mid z_i=k, \beta) \sim \mathcal{N}(\mu_k, \Sigma_k)$$

... then is it possible to make the claim that this empirical distribution over $$\alpha$$ has a regularising effect over the amount of clusters forming ...

Your analysis is correct, since the choice of $$\alpha$$ governs how many clusters will be allocated and whether the model will have a tendency to favor existing clusters or generate more clusters.

## References:

1. Gibbs sampling for fitting finite and infinite Gaussian mixture models - by Herman Kamper

2. A tutorial on Bayesian nonparametric models - by Gershman, Blei

• Thanks a lot for your reply. Just a few follow up comments. Perhaps a misunderstanding on my part, but my original question wasn't "is $P(\textbf{z} \mid \textbf{X}) = P(\textbf{z} \mid \alpha)$", but if $P(\textbf{z} \mid \alpha, \textbf{X}) = P(\textbf{z} \mid \alpha)$". Also in the following answer you state $P(z_i=k \mid \alpha)$ and seem to suggest it is equivalent to $P(\textbf{z} \mid \textbf{X})$? Perhaps I am not filling in the gaps because you are simplifying notation, as per your previous comment on my plate model, and I get lost in some of the conditioning?? – pche8701 Apr 24 at 7:26
• Also I am happy that we agree on the regularizing effect on the presence of $\alpha$ but would you know of any sources or references which go into more detail on this? In particular some sort of mathematical discussion demonstrating its regularising behaviour? Thank You! – pche8701 Apr 24 at 7:31
• @pche8701 Sorry for the confusion. I have clarified some notations – kedarps Apr 24 at 14:06
• As per the comment regarding 'regularization' I am not aware of specific literature. Will let you know if I find any. – kedarps Apr 24 at 14:07
• Thanks a lot for your responses. I'm just a little bit stuck still on the first question. I understand your idea, but when you look at equation (2) $P(\textbf{z} \mid \textbf{X}, \alpha, \beta) \propto P(\textbf{X} \mid \textbf{z}, \beta) \times P(\textbf{z} \mid \alpha)$, we see that the $\textbf{X}$ term only comes into play when considering the $\beta$ hyper parameter, not when $\alpha$ is involved, which goes back to the original Q "is $P(\textbf{z} \mid \alpha, \textbf{X}) = P(\textbf{z} \mid \alpha)$", since as your posterior suggests $\textbf{X}$ only effects $\beta$ and not $\alpha$. – pche8701 Apr 28 at 3:37