In this page of Murphy's 'Machine Learning: a Probabilistic Perspective' it's explained how to do Gibbs sampling on a Gaussian Mixture Model.

Reading this, I was trying to understand when to update parameters 'all together' and when to separate them: in Gibbs Sampling, you update one parameter at the time. According to this book, however, you update the 'multidimensional' means for each cluster 'all together'. That is, for each cluster k, it updates $[\mu_{k1}, \mu_{k2}] $ in one go. However, would it work if hypothetically I first update $\mu_{k1}$ and then $\mu_{k2}$?

I am asking to improve my understanding of this method. Thanks

enter image description here



in Gibbs Sampling, you update one parameter at the time

is not correct. Gibbs sampling is using a decomposition of the distribution of interest into conditional distributions for blocks of components of the vector $X$ to be simulated. These blocks can be of any dimension. Provided all components of $X$ belong to at least one of the blocks, and that there is no irreducibility issue with the conditionals, the associated Gibbs sampler is valid. Hence, if instead of simulating the vector $\boldsymbol \mu_k$ at once, as in (24.12), one simulates each component of $\boldsymbol \mu_k$ sequentially, conditional on everything else, this is also a correct implementation of the Gibbs sampler.

  • $\begingroup$ Thank you. After posting the question I kept searching on the internet and found the block sampling technique which clarified my confusion, but thank you for confirming it. Is it correct to say that updating one component at the time is the 'standard' implementation, though? $\endgroup$ – Vaaal Mar 26 '20 at 8:46
  • 1
    $\begingroup$ Erm, no, I do not think insisting upon 1D simulations is standard. The larger the block the faster the convergence in most cases. $\endgroup$ – Xi'an Mar 26 '20 at 8:59
  • $\begingroup$ @Xi'an Can you please see this question: stats.stackexchange.com/questions/498659/…? Thanks. $\endgroup$ – user261225 Nov 30 '20 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.