5
$\begingroup$

Suppose you have some complex model you want to sample from by Markov chain Monte Carlo. There are many types of situations where you can divide your variables into, say, two groups, and efficiently and exactly sample the variables in one group conditional on the others. That is, we can sample from

$ p(x | y)$

and

$ p(y | x)$

and so we would run our Gibbs sampler by repeatedly drawing $x$ from $p(x|y)$ and $y$ from $p(y | x)$.

Examples include the restricted Boltzmann machine, scale mixtures of Gaussians in certain image models, and some Bayesian models (one block for the state, and one for parameters). Anyway, in practice, being able to implement block Gibbs sampling seems to lead to much faster mixing of the Markov chain. But, are there any theoretical results on this? Presumably some conditions will need to be satisfied by the two distributions, which will lead to a bound on conductance, or a coupling of some sort. Thanks in advance!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
8
$\begingroup$

Much faster than what? Univariate Gibbs sampling?

The two stage Gibbs sampling is certainly the most studied type of Gibbs sampling starting with Tanner and Wong (1987, JASA). There is in particular a very achieved paper by Liu, Wong and Kong (1994, Biometrika), which shows that the correlation between the $X_t$'s (and the $Y_t$'s) is (a) positive and (b) going down to zero monotonically.

Blocked Gibbs sampling is usually more efficient than one-at-a-time Gibbs sampling but I do not know of a general result that would say so. In particular, augmenting the dimension with auxiliary variables may improve convergence, see the recent work of Xiao-Li Meng in JCGS as an illustration.

Here is an entry on this other forum that brings additional references.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks-- I think that Tanner and Wong's uniform boundedness condition and associated Theorem (Condition C, Theorem 3) pretty much constitute a valid answer. (And yep, I wanted to compare to univariate sampling.) $\endgroup$
    – user8434
    Commented Jan 11, 2012 at 13:50
  • 1
    $\begingroup$ I tried, of course, but the system won't allow me to currently, due to my insufficient reputation. $\endgroup$
    – user8434
    Commented Jan 12, 2012 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.