3
$\begingroup$

In this page of Murphy's 'Machine Learning: a Probabilistic Perspective' it's explained how to do Gibbs sampling on a Gaussian Mixture Model.

Reading this, I was trying to understand when to update parameters 'all together' and when to separate them: in Gibbs Sampling, you update one parameter at the time. According to this book, however, you update the 'multidimensional' means for each cluster 'all together'. That is, for each cluster k, it updates $[\mu_{k1}, \mu_{k2}] $ in one go. However, would it work if hypothetically I first update $\mu_{k1}$ and then $\mu_{k2}$?

I am asking to improve my understanding of this method. Thanks

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

First

in Gibbs Sampling, you update one parameter at the time

is not correct. Gibbs sampling is using a decomposition of the distribution of interest into conditional distributions for blocks of components of the vector $X$ to be simulated. These blocks can be of any dimension. Provided all components of $X$ belong to at least one of the blocks, and that there is no irreducibility issue with the conditionals, the associated Gibbs sampler is valid. Hence, if instead of simulating the vector $\boldsymbol \mu_k$ at once, as in (24.12), one simulates each component of $\boldsymbol \mu_k$ sequentially, conditional on everything else, this is also a correct implementation of the Gibbs sampler.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you. After posting the question I kept searching on the internet and found the block sampling technique which clarified my confusion, but thank you for confirming it. Is it correct to say that updating one component at the time is the 'standard' implementation, though? $\endgroup$
    – Vaaal
    Commented Mar 26, 2020 at 8:46
  • 1
    $\begingroup$ Erm, no, I do not think insisting upon 1D simulations is standard. The larger the block the faster the convergence in most cases. $\endgroup$
    – Xi'an
    Commented Mar 26, 2020 at 8:59
  • $\begingroup$ @Xi'an Can you please see this question: stats.stackexchange.com/questions/498659/…? Thanks. $\endgroup$
    – user261225
    Commented Nov 30, 2020 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.