# When can the collapsed Gibbs sampler be implemented?

I understand Gibbs sampling is a means of statistics inference, and it seems that sometimes certain variables can be integrated out in the sampling process, known as collapsed Gibbs sampling. I really want to know in what circumstances the collapsed Gibbs sampling can be applied, and which variables can be integrated out?

I did some search on Google, and it appears there are no detailed explanation on it. Though there are some papers on applying collapsed Gibbs sampling on Latent Dirichlet Allocation (LDA), I am no expert of MCMC and have no idea what LDA is, so it may be hard for me to read those papers. Can someone answer this question and it would be better, provide some examples? Much appreciated!

Suppose the density of the target distribution is $f(x, y, z)$ and the goal is to draw samples from this distribution. This is often difficult because the density is only known upto a normalizing constant. That is, $f$ is not known entirely but the function $g$ below is $$f(x,y,z) \propto g(x,y,z)$$

The usual Gibbs sampler would require obtaining all three full conditionals and cycling through them. That is, for the $t$th sample

1. Draw $x_t$ from $x \mid y_{t-1}, z_{t-1}$
2. Draw $y_t$ from $y \mid x_t, z_{t-1}$
3. Draw $z_t$ from $z \mid x_t, y_t$

Naturally, in order to implement this Gibbs sampler, it must be straightforward to sample from all three full conditionals using standard software. Often, obtaining these full conditionals is not possible, in which case a full Gibbs sampler is not implementable to begin with. Then $\{(x_t, y_t, z_t)\}_t$ forms a Markov chain with invariant distribution having density $f(x,y,z)$.

To implement a collapsed Gibbs sampler, you first integrate either $x$, $y$, or $z$ out and then run a Gibbs sampler on the marginal of the remaining two. Suppose, you integrate $z$ out. Now that target density is $$f(x,y) = \int f(x, y, z) \,dz\,.$$

Note that if a Gibbs sampler like the one above is possible, then it is straightforward to integrate out $z$, since \begin{align*} f(x,y) & = \int f(x, y, z) \,dz\\ & = c \int g(x,y,z) dz\\ & \overset{(a)}{=} c'g(x,y) \int f(z\mid x,y) dz\\ & \propto g(x,y) \end{align*} where the equality $(a)$ holds because a closed form expression of $f(z \mid x,y)$ is attainable since a usual Gibbs sampler (as described above) is possible.

Now, if it is possible to run a Gibbs sampler with $f(x,y)$ as the invariant distribution, then the resulting sampler is a collapsed Gibbs sampler. So if $f(x\mid y)$ and $f(y \mid x)$ are available in closed form so that for the $t$th sample

1. Draw $x_t$ from $x_t \mid y_{t-1}$
2. Draw $y_t$ from $y_t \mid x_t$

Then $\{(x_t, y_t)\}_t$ is a Markov chain with stationary distribution $f(x,y)$. Note that since $f(x,y,z) = f(z \mid x,y) f(x,y)$, the samples $z_t$ can be obtained drawing from density with distribution $f(z \mid x_t, y_t)$. Running this Gibbs sampler depends heavily on whether once can sample from $x\mid y$ and $y \mid x$.

To summarize, a collapsed Gibbs sampler can be implemented if for a group of three variables $x,y$ and $z$, one can sample from $x|y$, $y|x$ and $z|x,y$. One can choose which variable to integrate out based on which conditional distributions are available.

Some books on these topics are:

Monte Carlo Statistical Methods by C. Robert : Good for an introduction to Gibbs sampling

Monte Carlo Strategies in Scientific Computing by J. Liu : Has some discussion of collapsed Gibbs samplers (I think).

• Thanks for answering my question. I am a little confused. Regarding "where the equality (a) holds because a closed form expression of f(z∣x,y) is attainable since a full Gibbs sampler is possible.", which full Gibbs sampler are you referring to? – Steve Yang Mar 8 '18 at 1:11
• One more question, what is the invariant distribution of a Markov chain? Is it the same thing as the stationary distribution? – Steve Yang Mar 8 '18 at 1:26
• I realize that you are an expert in MCMC and Bayesian computation. I just start to learn MCMC and Bayesian statistics, it would be better if you could provide me some great books regarding to these topic. Thanks again! – Steve Yang Mar 8 '18 at 1:33
• @SteveYang I have added some references. And yes, invariant distribution is another name for stationary distribution. By full Gibbs sampler, I meant the usual Gibbs sampler I described in the beginning. – Greenparker Mar 8 '18 at 10:43