In Rasmussen's paper it is introduced a Gibbs sampler to make inference about a standard Gaussian Mixture Model.

To simplify, assume the 1-d case with basic hierarchical structure, that is:

$x_i|z_i= k \sim N(\mu_k,\sigma^2_k)$

with mixing weight $\pi_k$ and standard conjugate priors:

$\mu_k \sim N(\mu_0,\sigma^2_0) \quad \quad \sigma^2_k \sim InvGamma(\gamma,\beta) \quad \pi_k \sim Dir(\alpha/k,...,\alpha/k)$

Now taking into account the latent variables $z_i$, in order to implement the Gibbs Sampler I would write first the full conditional distribution of $\mu_k$,$\sigma^2_k$,$\pi_k$,$z_i$, while the paper completely ignore $\pi_k$ and makes inference about the mixing weight through the inference of the indicator variables.

It is somehow related to the Collapsed Gibbs?

  • $\begingroup$ It is because the $\pi_k$ are independent from the observations and the other parameters given the $z_i$'. $\endgroup$ – Xi'an Aug 26 '19 at 19:34
  • $\begingroup$ Could you please elaborate the answer? In the paper, when calculating $p(z_i=j |z_{-i},\alpha)$, $\pi$ is integrated out, and obviously the other full conditional distributions do not depend on $\pi$, is it somehow related? $\endgroup$ – momomi Aug 27 '19 at 7:20
  • $\begingroup$ I would like to refer you to my earlier paper with Diebolt, in JRSS B 1994, written on 1989, where we develop the first (?) Gibbs sampler for a mixture model. Or to the first edition of our Monte Carlo Statistical Methods with George Casella and a full chapter on this topic. Or yet to our book Bayesian Core with Jean-Michel Marin. My point is that this is extremely well treated in the literature. $\endgroup$ – Xi'an Aug 27 '19 at 7:29
  • $\begingroup$ I was reading algorithm 6.3 at page 156 of Bayesian Core, which should be exactly the case of my example. The Gibbs Sampler iterates between all the full conditionals including $p|z$. That's what I can't get, because this step is ignored in Rasmussen's paper. I'm sorry but I do not understand your first answer to my question. Actually the presentation of your book Bayesian Core is what I would have done without looking at Rasmussen's paper, according to my general (and basic) knowledge about Gibbs Sampler. $\endgroup$ – momomi Aug 27 '19 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.