When can the collapsed Gibbs sampler be implemented? I understand Gibbs sampling is a means of statistics inference, and it seems that sometimes certain variables can be integrated out in the sampling process, known as collapsed Gibbs sampling. I really want to know in what circumstances the collapsed Gibbs sampling can be applied, and which variables can be integrated out?
I did some search on Google, and it appears there are no detailed explanation on it. Though there are some papers on applying collapsed Gibbs sampling on Latent Dirichlet Allocation (LDA), I am no expert of MCMC and have no idea what LDA is, so it may be hard for me to read those papers. Can someone answer this question and it would be better, provide some examples? Much appreciated!
 A: Suppose the density of the target distribution is $f(x, y, z)$ and the goal is to draw samples from this distribution. This is often difficult because the density is only known upto a normalizing constant. That is, $f$ is not known entirely but the function $g$ below is
$$f(x,y,z) \propto g(x,y,z) $$ 
The usual Gibbs sampler would require obtaining all three full conditionals and cycling through them. That is, for the $t$th sample


*

*Draw $x_t$ from $x \mid y_{t-1}, z_{t-1}$

*Draw $y_t$ from $y \mid x_t, z_{t-1}$

*Draw $z_t$ from $z \mid x_t, y_t$


Naturally, in order to implement this Gibbs sampler, it must be straightforward to sample from all three full conditionals using standard software. Often, obtaining these full conditionals is not possible, in which case a full Gibbs sampler is not implementable to begin with. Then $\{(x_t, y_t, z_t)\}_t$ forms a Markov chain with invariant distribution having density $f(x,y,z)$. 
To implement a collapsed Gibbs sampler, you first integrate either $x$, $y$, or $z$ out and then run a Gibbs sampler on the marginal of the remaining two. Suppose, you integrate $z$ out. Now that target density is
$$f(x,y) = \int f(x, y, z) \,dz\,.$$
Note that if a Gibbs sampler like the one above is possible, then it is straightforward to integrate out $z$, since 
\begin{align*}
f(x,y) & = \int f(x, y, z) \,dz\\
& = c \int g(x,y,z) dz\\
& \overset{(a)}{=} c'g(x,y) \int f(z\mid x,y) dz\\
& \propto g(x,y)
\end{align*}
where the equality $(a)$ holds because a closed form expression of $f(z \mid x,y)$ is attainable since a usual Gibbs sampler (as described above) is possible. 
Now, if it is possible to run a Gibbs sampler with $f(x,y)$ as the invariant distribution, then the resulting sampler is a collapsed Gibbs sampler. So if $f(x\mid y)$ and $f(y \mid x)$ are available in closed form so that for the $t$th sample


*

*Draw $x_t$ from $x_t \mid y_{t-1}$

*Draw $y_t$ from $y_t \mid x_t$


Then $\{(x_t, y_t)\}_t$ is a Markov chain with stationary distribution $f(x,y)$. Note that since $f(x,y,z) = f(z \mid x,y) f(x,y)$, the samples $z_t$ can be obtained drawing from density with distribution $f(z \mid x_t, y_t)$. Running this Gibbs sampler depends heavily on whether once can sample from $x\mid y$ and $y \mid x$. 
To summarize, a collapsed Gibbs sampler can be implemented if for a group of three variables $x,y$ and $z$, one can sample from $x|y$, $y|x$ and $z|x,y$. One can choose which variable to integrate out based on which conditional distributions are available.
Some books on these topics are:
Monte Carlo Statistical Methods by C. Robert : Good for an introduction to Gibbs sampling
Monte Carlo Strategies in Scientific Computing by J. Liu : Has some discussion of collapsed Gibbs samplers (I think).
