2
$\begingroup$

Suppose I have a Dirichlet Process Mixture model defined as follows:

$\alpha \sim G(a,b)\\ \pi|\alpha \sim \text{Dir}(\alpha)\\ z|\pi \sim \text{Cat}(\pi)\\ $

where $G$ is just a standard Gamma distribution, and

$ \mu,\Sigma \sim NIW(\beta) $ where $NIW$ stand for Normal Inverse Wishart.

Ultimately the plate model looks like:

enter image description here

Upon doing some probabilistic maniuplation I arrive at the following terms I would like some clarification on: $p(z|X),p(z|X,\alpha),p(X|z)$. I am just wondering how to interpret these.

  • Is $p(z|X,\alpha) = p(z|\alpha)$? It seems that $X$ is already implicitly used in the $p(z|\alpha)$ expression as the `number of terms allocated to a cluster' in the sense of the Chinese Restaurant Process, when considering $p(z|\alpha)$ by itself, so does conditioning on $X$ change anything?

  • How should I interpret the expressions $p(z|X)$ and $p(X|z)$? It feels strange because $z$ just acts like a `switch' or index for this Bayes Net, so interpreting either direction of these feels strange.

  • In addition, if I add a (Gamma) prior distribution over the $\alpha$ term, with a likelihood term similar to that suggested by Escobar & West 1995 (equations 13 / 14) then is it possible to make the claim that this empirical distribution over $\alpha$ has a regularising effect over the amount of clusters forming (or in the cluster allocations for each point, that is $p(z_i = k | X, \beta, \alpha, z_{\lnot i}$).

$\endgroup$
  • $\begingroup$ Why do you have an arrow between $\mu_K$ and $\Sigma_K$ nodes? $\endgroup$ – kedarps Apr 23 at 6:15
  • $\begingroup$ This is common in normal inverse wishart models where the output of $\Sigma_k$ can model the variance over the $\mu_k$, see the wiki en.wikipedia.org/wiki/Normal-inverse-Wishart_distribution $\endgroup$ – pche8701 Apr 23 at 6:17
  • $\begingroup$ Yes, I am aware of that. But you draw both $\mu_K$ and $\Sigma_K$ from NIW. With the arrow you are implying that a draw of $\mu_K$ depends on the draw of $\Sigma_K$ -- which is not true. Hence there is no need for an arrow between the two nodes. $\endgroup$ – kedarps Apr 23 at 6:24
  • $\begingroup$ I am probably wrong, but I just always assumed that the arrows point in the direction of dependency, and $\mu_k$ does depend on $\Sigma_k$. Is this an incorrect understanding? $\endgroup$ – pche8701 Apr 23 at 6:34
  • $\begingroup$ This is a minor point and not so important from the context of your question in the post. You are correct, that if you are drawing $\mu_K$ and $\Sigma_K$ separately, you would draw $\Sigma_K$ from Inverse-Wishart distribution followed by $\mu_K$ from a Normal distribution. Since you are specifying that $(\mu_K, \Sigma_K)$ are drawn from $NIW(\beta)$, you can just skip the arrow between the two to simplify the notation. $\endgroup$ – kedarps Apr 23 at 6:45
1
+50
$\begingroup$

Let's review the generative process assumed to generate data using GMM with infinite (i.e. non-fixed) number of clusters.

  1. First we need to choose a cluster assignment. Using a Chinese restaurant process, we assume that there are currently $K$ clusters, but we also have a probability to assign an observation to a new cluster $K+1$. This way we do not need to fix $K$ a priori,

$$ \begin{equation}\tag{1}\label{eqn1} P(z_i=k\mid \alpha) = \begin{cases} \frac{N_k}{N+\alpha-1} & , k \in [1, K]\\ \frac{\alpha}{N+\alpha-1} & , k = k_{new}=K+1\\ \end{cases} \end{equation} $$

  1. Given a cluster assignment, we can generate an observation from the corresponding Gaussian with parameters: $\mathcal{N}(\mu_k, \Sigma_k)$

For clustering, we need to determine the probability of assigning an observation to a cluster, which can be done using Bayes rule as follows,

$$\tag{2}\label{eqn2} P(\textbf{z} \mid \textbf{X}, \alpha, \beta) \propto P(\textbf{X} \mid \textbf{z}, \beta) \times P(\textbf{z} \mid \alpha) $$

If we are doing MCMC sampling, the above equation can be written as,

$$\tag{3}\label{eqn3} P(z_i=k \mid \textbf{z}_{-i}, \textbf{X}, \alpha, \beta) \propto P(x_i \mid \textbf{X}_{-i}, z_i=k, \textbf{z}_{-i}, \beta) \times P(z_i=k \mid \textbf{z}_{-i}, \alpha) $$


Answering your questions:

Is $P(\textbf{z} \mid \textbf{X}, \alpha, \beta) = P(\textbf{z} \mid \alpha)$?

No they are not. $P(\textbf{z} \mid \textbf{X}, \alpha, \beta)$ is the posterior evaluated using equation $(\ref{eqn2})$ and you get $P(\textbf{z} \mid \alpha)$ by integrating out $\pi$ as follows:

$$ P(\textbf{z} \mid \alpha) = \int_{\pi}P(\textbf{z} \mid \mathbb{\pi}) \: p(\mathbb{\pi} \mid \alpha) \: d\pi $$

How should I interpret the expressions $P(\textbf{z} \mid \textbf{X})$ and $P(\textbf{X} \mid \textbf{z})$?

The former term is the posterior distribution $[$ i.e. which is (\ref{eqn2}) ignoring $\alpha$ and $\beta$ $]$ of assigning clusters to the data, the latter is the likelihood of the data. The posterior is calculated using the two terms:

  • $P(z_i=k \mid \alpha)$ is given by equation $(\ref{eqn1})$
  • $P(x_i \mid z_i=k, \beta) \sim \mathcal{N}(\mu_k, \Sigma_k)$

... then is it possible to make the claim that this empirical distribution over $\alpha$ has a regularising effect over the amount of clusters forming ...

Your analysis is correct, since the choice of $\alpha$ governs how many clusters will be allocated and whether the model will have a tendency to favor existing clusters or generate more clusters.


References:

  1. Gibbs sampling for fitting finite and infinite Gaussian mixture models - by Herman Kamper

  2. A tutorial on Bayesian nonparametric models - by Gershman, Blei

$\endgroup$
  • $\begingroup$ Thanks a lot for your reply. Just a few follow up comments. Perhaps a misunderstanding on my part, but my original question wasn't "is $P(\textbf{z} \mid \textbf{X}) = P(\textbf{z} \mid \alpha)$", but if $P(\textbf{z} \mid \alpha, \textbf{X}) = P(\textbf{z} \mid \alpha)$". Also in the following answer you state $P(z_i=k \mid \alpha)$ and seem to suggest it is equivalent to $P(\textbf{z} \mid \textbf{X})$? Perhaps I am not filling in the gaps because you are simplifying notation, as per your previous comment on my plate model, and I get lost in some of the conditioning?? $\endgroup$ – pche8701 Apr 24 at 7:26
  • $\begingroup$ Also I am happy that we agree on the regularizing effect on the presence of $\alpha$ but would you know of any sources or references which go into more detail on this? In particular some sort of mathematical discussion demonstrating its regularising behaviour? Thank You! $\endgroup$ – pche8701 Apr 24 at 7:31
  • $\begingroup$ @pche8701 Sorry for the confusion. I have clarified some notations $\endgroup$ – kedarps Apr 24 at 14:06
  • $\begingroup$ As per the comment regarding 'regularization' I am not aware of specific literature. Will let you know if I find any. $\endgroup$ – kedarps Apr 24 at 14:07
  • $\begingroup$ Thanks a lot for your responses. I'm just a little bit stuck still on the first question. I understand your idea, but when you look at equation (2) $P(\textbf{z} \mid \textbf{X}, \alpha, \beta) \propto P(\textbf{X} \mid \textbf{z}, \beta) \times P(\textbf{z} \mid \alpha)$, we see that the $\textbf{X}$ term only comes into play when considering the $\beta$ hyper parameter, not when $\alpha$ is involved, which goes back to the original Q "is $P(\textbf{z} \mid \alpha, \textbf{X}) = P(\textbf{z} \mid \alpha)$", since as your posterior suggests $\textbf{X}$ only effects $\beta$ and not $\alpha$. $\endgroup$ – pche8701 Apr 28 at 3:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.