Bayesian mixture model joint posterior

Question

I am just starting to learn about bayesian mixture models. There is a few clarifications that I want to make which I am not sure myself. The graphical model below describes a gaussian mixture model where $\pi, \Psi, \Sigma, \mu, \boldsymbol{Z}$ are the latent unknowns that we would like to infer about the mixture model. $\mu,\Sigma, Z$ are vectors since we are parameterising by $K$ clusters.

From the model described, am i right to say that the joint posterior explicitly is $$p(\boldsymbol{Z},\Sigma,\mu, \pi | \alpha, \mu_0, \Sigma_0, \Psi, \{X_n\}_{i=1}^N) = p(\pi|\alpha)\prod_{k=1}^Kp(\mu_k|\mu_0,\Sigma_0)p(\Sigma_k|\Psi)\prod_{i=1}^Np(z_i|\pi)p(x_i|z_i, \mu,\Sigma)$$

From this full joint posterior, how do i use conditional independence to compute the conditional posterior distributions for each variable to do gibbs sampling ?..

Answer 1

I think I can help. I'll give a partial answer and if you need more then let me know.

First I'd like to simplify the notation a bit. Let $X = (x_1, \ldots, x_N)$ denote the data, which is in parallel with $Z = (z_1, \ldots, z_N)$. Also I will suppress the known parameters in the priors for the unknowns $(\pi,Z,\mu,\Sigma)$. With these changes, the joint posterior can be expressed as proportional to the joint distribution \begin{equation} p(\pi,Z,\mu,\Sigma|X) \propto p(\pi,Z,\mu,\Sigma,X) . \end{equation} The Gibbs sampler involves cycling through the full conditional distributions: \begin{align} p(\pi|X,Z,\mu,\Sigma) &\propto p(\pi,Z,\mu,\Sigma,X) \\ p(Z|X,\pi,\mu,\Sigma) &\propto p(\pi,Z,\mu,\Sigma,X) \\ p(\mu|X,\pi,Z,\Sigma) &\propto p(\pi,Z,\mu,\Sigma,X) \\ p(\Sigma|X,\pi,Z,\mu) &\propto p(\pi,Z,\mu,\Sigma,X) , \end{align} where each conditional distribution is proportional to the joint distribution.

Let's begin with the distribution for $\pi$. We need to collect all the factors that involve $\pi$ and ignore the rest. Therefore, \begin{equation} p(\pi|X,Z,\mu,\Sigma) \propto p(\pi|\alpha)\,\prod_{i=1}^N p(z_i|\pi) . \end{equation} That's the main idea. You can apply it to the other unknowns.

I could stop here, but I think it will help to provide some interpretation for some of these symbols. In particular, \begin{equation} z_i \in \{1, \ldots, K\} \end{equation} is a latent classification variable the indicates which mixture component $x_i$ belongs to. Therefore, \begin{equation} p(x_i|z_i,\mu,\Sigma) = \textsf{N}(x_i|\mu_{z_i}, \Sigma_{z_i}) . \end{equation} In addition, \begin{equation} p(z_i|\pi) = \textsf{Categorical}(z_i|\pi) = \prod_{k=1}^K \pi_k^{1(z_i=k)} \end{equation} where \begin{equation} 1(z_i=k) = \begin{cases} 1 & z_i = k \\ 0 & \text{otherwise} \end{cases} . \end{equation} Just to be clear, $p(z_i = k|\pi) = \pi_k$. So $\pi$ is a vector of $K$ probabilities that sum to one and (very likely, but not necessarily) \begin{equation} p(\pi|\alpha) = \textsf{Dirichlet}(\pi|\alpha) \propto \prod_{k=1}^K \pi_k^{\alpha_k-1} . \end{equation}

You now have the prior and the conditional likelihood for $\pi$, which enough to figure out what the conditional posterior for $\pi$ is. So I will stop here.