2
$\begingroup$

Suppose I have a Dirichlet Process Mixture model defined as follows:

$\alpha \sim G(a,b)\\ \pi|\alpha \sim \text{Dir}(\alpha)\\ z|\pi \sim \text{Cat}(\pi)\\ $

where $G$ is just a standard Gamma distribution, and

$ \mu,\Sigma \sim NIW(\beta) $ where $NIW$ stand for Normal Inverse Wishart.

Ultimately the plate model looks like:

enter image description here

Upon doing some probabilistic maniuplation I arrive at the following terms I would like some clarification on: $p(z|X),p(z|X,\alpha),p(X|z)$. I am just wondering how to interpret these.

  • Is $p(z|X,\alpha) = p(z|\alpha)$? It seems that $X$ is already implicitly used in the $p(z|\alpha)$ expression as the `number of terms allocated to a cluster' in the sense of the Chinese Restaurant Process, when considering $p(z|\alpha)$ by itself, so does conditioning on $X$ change anything?

  • How should I interpret the expressions $p(z|X)$ and $p(X|z)$? It feels strange because $z$ just acts like a `switch' or index for this Bayes Net, so interpreting either direction of these feels strange.

  • In addition, if I add a (Gamma) prior distribution over the $\alpha$ term, with a likelihood term similar to that suggested by Escobar & West 1995 (equations 13 / 14) then is it possible to make the claim that this empirical distribution over $\alpha$ has a regularising effect over the amount of clusters forming (or in the cluster allocations for each point, that is $p(z_i = k | X, \beta, \alpha, z_{\lnot i}$).

$\endgroup$
7
  • $\begingroup$ Why do you have an arrow between $\mu_K$ and $\Sigma_K$ nodes? $\endgroup$
    – kedarps
    Commented Apr 23, 2019 at 6:15
  • $\begingroup$ This is common in normal inverse wishart models where the output of $\Sigma_k$ can model the variance over the $\mu_k$, see the wiki en.wikipedia.org/wiki/Normal-inverse-Wishart_distribution $\endgroup$ Commented Apr 23, 2019 at 6:17
  • $\begingroup$ Yes, I am aware of that. But you draw both $\mu_K$ and $\Sigma_K$ from NIW. With the arrow you are implying that a draw of $\mu_K$ depends on the draw of $\Sigma_K$ -- which is not true. Hence there is no need for an arrow between the two nodes. $\endgroup$
    – kedarps
    Commented Apr 23, 2019 at 6:24
  • $\begingroup$ I am probably wrong, but I just always assumed that the arrows point in the direction of dependency, and $\mu_k$ does depend on $\Sigma_k$. Is this an incorrect understanding? $\endgroup$ Commented Apr 23, 2019 at 6:34
  • $\begingroup$ This is a minor point and not so important from the context of your question in the post. You are correct, that if you are drawing $\mu_K$ and $\Sigma_K$ separately, you would draw $\Sigma_K$ from Inverse-Wishart distribution followed by $\mu_K$ from a Normal distribution. Since you are specifying that $(\mu_K, \Sigma_K)$ are drawn from $NIW(\beta)$, you can just skip the arrow between the two to simplify the notation. $\endgroup$
    – kedarps
    Commented Apr 23, 2019 at 6:45

1 Answer 1

1
+50
$\begingroup$

Let's review the generative process assumed to generate data using GMM with infinite (i.e. non-fixed) number of clusters.

  1. First we need to choose a cluster assignment. Using a Chinese restaurant process, we assume that there are currently $K$ clusters, but we also have a probability to assign an observation to a new cluster $K+1$. This way we do not need to fix $K$ a priori,

$$ \begin{equation}\tag{1}\label{eqn1} P(z_i=k\mid \alpha) = \begin{cases} \frac{N_k}{N+\alpha-1} & , k \in [1, K]\\ \frac{\alpha}{N+\alpha-1} & , k = k_{new}=K+1\\ \end{cases} \end{equation} $$

  1. Given a cluster assignment, we can generate an observation from the corresponding Gaussian with parameters: $\mathcal{N}(\mu_k, \Sigma_k)$

For clustering, we need to determine the probability of assigning an observation to a cluster, which can be done using Bayes rule as follows,

$$\tag{2}\label{eqn2} P(\textbf{z} \mid \textbf{X}, \alpha, \beta) \propto P(\textbf{X} \mid \textbf{z}, \beta) \times P(\textbf{z} \mid \alpha) $$

If we are doing MCMC sampling, the above equation can be written as,

$$\tag{3}\label{eqn3} P(z_i=k \mid \textbf{z}_{-i}, \textbf{X}, \alpha, \beta) \propto P(x_i \mid \textbf{X}_{-i}, z_i=k, \textbf{z}_{-i}, \beta) \times P(z_i=k \mid \textbf{z}_{-i}, \alpha) $$


Answering your questions:

Is $P(\textbf{z} \mid \textbf{X}, \alpha, \beta) = P(\textbf{z} \mid \alpha)$?

No they are not. $P(\textbf{z} \mid \textbf{X}, \alpha, \beta)$ is the posterior evaluated using equation $(\ref{eqn2})$ and you get $P(\textbf{z} \mid \alpha)$ by integrating out $\pi$ as follows:

$$ P(\textbf{z} \mid \alpha) = \int_{\pi}P(\textbf{z} \mid \mathbb{\pi}) \: p(\mathbb{\pi} \mid \alpha) \: d\pi $$

How should I interpret the expressions $P(\textbf{z} \mid \textbf{X})$ and $P(\textbf{X} \mid \textbf{z})$?

The former term is the posterior distribution $[$ i.e. which is (\ref{eqn2}) ignoring $\alpha$ and $\beta$ $]$ of assigning clusters to the data, the latter is the likelihood of the data. The posterior is calculated using the two terms:

  • $P(z_i=k \mid \alpha)$ is given by equation $(\ref{eqn1})$
  • $P(x_i \mid z_i=k, \beta) \sim \mathcal{N}(\mu_k, \Sigma_k)$

... then is it possible to make the claim that this empirical distribution over $\alpha$ has a regularising effect over the amount of clusters forming ...

Your analysis is correct, since the choice of $\alpha$ governs how many clusters will be allocated and whether the model will have a tendency to favor existing clusters or generate more clusters.


References:

  1. Gibbs sampling for fitting finite and infinite Gaussian mixture models - by Herman Kamper

  2. A tutorial on Bayesian nonparametric models - by Gershman, Blei

$\endgroup$
6
  • $\begingroup$ Thanks a lot for your reply. Just a few follow up comments. Perhaps a misunderstanding on my part, but my original question wasn't "is $P(\textbf{z} \mid \textbf{X}) = P(\textbf{z} \mid \alpha)$", but if $P(\textbf{z} \mid \alpha, \textbf{X}) = P(\textbf{z} \mid \alpha)$". Also in the following answer you state $P(z_i=k \mid \alpha)$ and seem to suggest it is equivalent to $P(\textbf{z} \mid \textbf{X})$? Perhaps I am not filling in the gaps because you are simplifying notation, as per your previous comment on my plate model, and I get lost in some of the conditioning?? $\endgroup$ Commented Apr 24, 2019 at 7:26
  • $\begingroup$ Also I am happy that we agree on the regularizing effect on the presence of $\alpha$ but would you know of any sources or references which go into more detail on this? In particular some sort of mathematical discussion demonstrating its regularising behaviour? Thank You! $\endgroup$ Commented Apr 24, 2019 at 7:31
  • $\begingroup$ @pche8701 Sorry for the confusion. I have clarified some notations $\endgroup$
    – kedarps
    Commented Apr 24, 2019 at 14:06
  • $\begingroup$ As per the comment regarding 'regularization' I am not aware of specific literature. Will let you know if I find any. $\endgroup$
    – kedarps
    Commented Apr 24, 2019 at 14:07
  • $\begingroup$ Thanks a lot for your responses. I'm just a little bit stuck still on the first question. I understand your idea, but when you look at equation (2) $P(\textbf{z} \mid \textbf{X}, \alpha, \beta) \propto P(\textbf{X} \mid \textbf{z}, \beta) \times P(\textbf{z} \mid \alpha)$, we see that the $\textbf{X}$ term only comes into play when considering the $\beta$ hyper parameter, not when $\alpha$ is involved, which goes back to the original Q "is $P(\textbf{z} \mid \alpha, \textbf{X}) = P(\textbf{z} \mid \alpha)$", since as your posterior suggests $\textbf{X}$ only effects $\beta$ and not $\alpha$. $\endgroup$ Commented Apr 28, 2019 at 3:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.