3
$\begingroup$

I am just starting to learn about bayesian mixture models. There is a few clarifications that I want to make which I am not sure myself. The graphical model below describes a gaussian mixture model where $\pi, \Psi, \Sigma, \mu, \boldsymbol{Z}$ are the latent unknowns that we would like to infer about the mixture model. $\mu,\Sigma, Z$ are vectors since we are parameterising by $K$ clusters.

enter image description here

From the model described, am i right to say that the joint posterior explicitly is $$p(\boldsymbol{Z},\Sigma,\mu, \pi | \alpha, \mu_0, \Sigma_0, \Psi, \{X_n\}_{i=1}^N) = p(\pi|\alpha)\prod_{k=1}^Kp(\mu_k|\mu_0,\Sigma_0)p(\Sigma_k|\Psi)\prod_{i=1}^Np(z_i|\pi)p(x_i|z_i, \mu,\Sigma)$$

From this full joint posterior, how do i use conditional independence to compute the conditional posterior distributions for each variable to do gibbs sampling ?..

$\endgroup$
  • $\begingroup$ Why not add "bayesian" as a tag? That way someone filtering on that tag will find your post. $\endgroup$ – mef Oct 17 at 15:51
  • $\begingroup$ Hey @mef thank you for your suggestion $\endgroup$ – calveeen Oct 17 at 16:21
  • $\begingroup$ Shouldn't the "sigma" in the final factor of the joint posterior be uppercase? $\endgroup$ – mef Oct 17 at 16:33
  • $\begingroup$ What is the difference between joint distribution vs joint posterior ? $\endgroup$ – calveeen Oct 18 at 1:43
  • $\begingroup$ I've modified my answer slightly. See if that helps. BTW, your "equation" for the posterior distribution should be a proportionality, not an equality. The right-hand side is the joint distribution and not the joint posterior. $\endgroup$ – mef Oct 18 at 16:58
4
$\begingroup$

I think I can help. I'll give a partial answer and if you need more then let me know.

First I'd like to simplify the notation a bit. Let $X = (x_1, \ldots, x_N)$ denote the data, which is in parallel with $Z = (z_1, \ldots, z_N)$. Also I will suppress the known parameters in the priors for the unknowns $(\pi,Z,\mu,\Sigma)$. With these changes, the joint posterior can be expressed as proportional to the joint distribution \begin{equation} p(\pi,Z,\mu,\Sigma|X) \propto p(\pi,Z,\mu,\Sigma,X) . \end{equation} The Gibbs sampler involves cycling through the full conditional distributions: \begin{align} p(\pi|X,Z,\mu,\Sigma) &\propto p(\pi,Z,\mu,\Sigma,X) \\ p(Z|X,\pi,\mu,\Sigma) &\propto p(\pi,Z,\mu,\Sigma,X) \\ p(\mu|X,\pi,Z,\Sigma) &\propto p(\pi,Z,\mu,\Sigma,X) \\ p(\Sigma|X,\pi,Z,\mu) &\propto p(\pi,Z,\mu,\Sigma,X) , \end{align} where each conditional distribution is proportional to the joint distribution.

Let's begin with the distribution for $\pi$. We need to collect all the factors that involve $\pi$ and ignore the rest. Therefore, \begin{equation} p(\pi|X,Z,\mu,\Sigma) \propto p(\pi|\alpha)\,\prod_{i=1}^N p(z_i|\pi) . \end{equation} That's the main idea. You can apply it to the other unknowns.

I could stop here, but I think it will help to provide some interpretation for some of these symbols. In particular, \begin{equation} z_i \in \{1, \ldots, K\} \end{equation} is a latent classification variable the indicates which mixture component $x_i$ belongs to. Therefore, \begin{equation} p(x_i|z_i,\mu,\Sigma) = \textsf{N}(x_i|\mu_{z_i}, \Sigma_{z_i}) . \end{equation} In addition, \begin{equation} p(z_i|\pi) = \textsf{Categorical}(z_i|\pi) = \prod_{k=1}^K \pi_k^{1(z_i=k)} \end{equation} where \begin{equation} 1(z_i=k) = \begin{cases} 1 & z_i = k \\ 0 & \text{otherwise} \end{cases} . \end{equation} Just to be clear, $p(z_i = k|\pi) = \pi_k$. So $\pi$ is a vector of $K$ probabilities that sum to one and (very likely, but not necessarily) \begin{equation} p(\pi|\alpha) = \textsf{Dirichlet}(\pi|\alpha) \propto \prod_{k=1}^K \pi_k^{\alpha_k-1} . \end{equation}

You now have the prior and the conditional likelihood for $\pi$, which enough to figure out what the conditional posterior for $\pi$ is. So I will stop here.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks @mef for the explanation. Is the posterior equation for the joint distribution that I have written correct ? I believe that $\alpha$ in the graphical model shouldn't be labelled a circle because it is not a variable we want to infer about $\endgroup$ – calveeen Oct 18 at 1:26
  • $\begingroup$ Sorry for the comment above, I did not phrase it correctly. I wanted to know whether the joint distribution for the mixture model is correctly written or have I missed something. $\endgroup$ – calveeen Oct 18 at 1:35
  • $\begingroup$ Also, the reason why I want to know how to write the joint distribution for finite mixture model is because I want to be able to write it for the infinite case. In many of the materials in the internet on DPMM, they do not show the full joint posterior but rather say that the $\pi$'s are integrated out of the model, leaving only the latent variable assignments $Z$ and cluster parameters $\theta$. If the joint distribution for the current finite mixture model is correctly written, then I can just replace $k$ in above with $\infty$ $\endgroup$ – calveeen Oct 18 at 1:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.