# Gibbs Sampler for GMM

In Rasmussen's paper it is introduced a Gibbs sampler to make inference about a standard Gaussian Mixture Model.

To simplify, assume the 1-d case with basic hierarchical structure, that is:

$$x_i|z_i= k \sim N(\mu_k,\sigma^2_k)$$

with mixing weight $$\pi_k$$ and standard conjugate priors:

$$\mu_k \sim N(\mu_0,\sigma^2_0) \quad \quad \sigma^2_k \sim InvGamma(\gamma,\beta) \quad \pi_k \sim Dir(\alpha/k,...,\alpha/k)$$

Now taking into account the latent variables $$z_i$$, in order to implement the Gibbs Sampler I would write first the full conditional distribution of $$\mu_k$$,$$\sigma^2_k$$,$$\pi_k$$,$$z_i$$, while the paper completely ignore $$\pi_k$$ and makes inference about the mixing weight through the inference of the indicator variables.

It is somehow related to the Collapsed Gibbs?

• It is because the $\pi_k$ are independent from the observations and the other parameters given the $z_i$'. – Xi'an Aug 26 '19 at 19:34
• Could you please elaborate the answer? In the paper, when calculating $p(z_i=j |z_{-i},\alpha)$, $\pi$ is integrated out, and obviously the other full conditional distributions do not depend on $\pi$, is it somehow related? – momomi Aug 27 '19 at 7:20
• I would like to refer you to my earlier paper with Diebolt, in JRSS B 1994, written on 1989, where we develop the first (?) Gibbs sampler for a mixture model. Or to the first edition of our Monte Carlo Statistical Methods with George Casella and a full chapter on this topic. Or yet to our book Bayesian Core with Jean-Michel Marin. My point is that this is extremely well treated in the literature. – Xi'an Aug 27 '19 at 7:29
• I was reading algorithm 6.3 at page 156 of Bayesian Core, which should be exactly the case of my example. The Gibbs Sampler iterates between all the full conditionals including $p|z$. That's what I can't get, because this step is ignored in Rasmussen's paper. I'm sorry but I do not understand your first answer to my question. Actually the presentation of your book Bayesian Core is what I would have done without looking at Rasmussen's paper, according to my general (and basic) knowledge about Gibbs Sampler. – momomi Aug 27 '19 at 9:03