0
$\begingroup$

I am studying MCMC and in the book I'm reading there is this example on Gibbs algorithm for inferring the posterior of a gaussian mixture. I understand how the algorithm works and the fact that its convenience relies in the simplicity of sampling from the full conditionals for the parameters, however I was wondering whether and why this method should be preferable to others that were introduced in previous chapters and with which I am less familiar (like Variational methods, or others I don't know).

Furthermore, the book starts by explicitly writing the full joint $p(x,z,\mu,\Sigma, \pi)$ for the gmm (assuming semi-conjugate prior), am I right to say that even knowing this explicitly, sampling directly (using for instance rejection sampling) is inefficient because of the curse of dimensionality, while Gibbs suffer far less from this phenomenon? Is this the only reason why Gibbs is to be preferred?

Any help on understanding pros and cons of the various existing methods for inferring the posterior is well accepted.

$\endgroup$
1
  • 1
    $\begingroup$ Gibbs sampling was introduced by Geman and Geman (1984) as a way to fight the curse of dimensionality. Devising a random walk Metropolis-Hastings algorithm in dimension $3k-1$ for a $k$ component Normal mixture is quite the challenge. $\endgroup$
    – Xi'an
    Feb 8, 2022 at 11:37

1 Answer 1

1
$\begingroup$

Gibbs sampling is possibly the first MCMC algorithm implemented for mixture models (Gelber, Gelman and Goldhirsch, 1989), inspired from the data augmentation of Tanner and Wong (1987) and ultimately from the EM algorithm. It takes advantage of the latent variable structure in producing nice, low dimension, and natural conditionals that can be simulated quite efficiently, much more than a default MCMC algorithm like random walk Metropolis-Hastings.

However, here is a slide from my MCMC course, where I illustrate the potential pitfall of using plain Gibbs for a mixture of two Gaussians with unknown means, namely that it may get trapped in a local mode because the latent variables are practically if not theoretically stuck at a fixed value.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.