slice sampling within a Gibbs sampler

Question

Questions

My questions are:

1. Is the following slice-sampling-within-Gibbs approach valid?
2. Is there a good reference out there that uses, or better yet, justifies it?

Context

I'm trying to sample from a high-dimensional joint distribution, which I'll simplify to $p(x, y)$. Let's say $x$ and $y$ are scalars for now. I would like to use a Gibbs sampler to draw samples. I.e., I would like to iterate:

1. Sample $x$ from its full conditional, $p(x | y)$.
2. Sample $y$ from its full conditional, $p(y | x)$.

Unfortunately, I can't sample from $p(x | y)$ or $p(y | x)$ directly. In my case, I have also tried replacing these Gibbs steps with Metropolis-Hastings and rejection sampling, but results were poor. I have had much better results replacing each Gibbs step with a univariate slice sampler. (See Radford Neal's 2003 paper for an explanation of slice sampling.) Now, I do a slice-sampling-within-Gibbs algorithm.

Slice-sampling-within-Gibbs

1. Use univariate slice sampling to draw $x$ from $p(x | y)$.
2. Use a separate and independent univariate slice sampler to draw $y$ from $p(y | x)$.

In case there is any confusion with multivariate slice sampling, what I specifically mean is:

1. Sample $u$ from Uniform(0, $p(x|y)$).
2. Sample $x$ from Uniform$\{x : u < p(x | y)\}$.
3. Sample $v$ from Uniform(0, $p(y|x)$).
4. Sample $y$ from Uniform$\{y : v < p(y | x)\}$.

where the uniforms in steps 2 and 4 are not known exactly. (Also, I don't need the samples of $u$ or $v$, so I discard those.)

Answer 1

I found two references. This one details the algorithm, but the publicly-available pages that I could see on Google Books don't prove that it works.

@inbook{cruz,
    Author = {Cruz, Marcelo G. and Peters, Gareth W. and Shevchenko, Pavel V.},
    Chapter = {7.6.2: Generic univariate auxiliary variable Gibbs sampler: slice sampler},
    Publisher = {Wiley},
    Title = {Fundamental Aspects of Operational Risk and Insurance Analytics: A Handbook of Operational Risk},
    Year = {2015}}

Another one, also partially available on Google Books for free, seems to allude to slice-sampling-within-Gibbs.

@inbook{banerjee,
    Author = {Banerjee, Sudipto and Carlin, Bradley P. and Gelfand, Alan E. },
    Chapter = {9.4.1: Regression in the Gaussian case},
    Edition = {2nd},
    Publisher = {CRC Press},
    Title = {Hierarchical Modeling and Analysis for Spatial Data},
    Year = {2015}}

I agree, it would be nice to find solid proof of validity, preferably in a good journal.

EDIT: Even Gelman's famous "Bayesian Data Analysis" (3rd ed) mentions the idea. In Section 12.3: Further extensions to Gibbs and Metropolis, under the "Slice sampling" heading, the end of the first paragraph says

Slice sampling refers to the application of iterative simulation algorithms on this uniform distribution. The details of implementing an effective slice sampling procedure can be complicated, but the method can be applied in great generality and can be especially useful for sampling one-dimensional conditional distributions in a Gibbs sampling structure.

Neal's famous 2003 slice sampling paper is where I think it was first suggested. The first paragraph of Section 4 says

Slice sampling is simplest when only one (real-valued) variable is being updated. This will of course be the case when the distribution of interest is univariate, but more typically, the single-variable slice sampling methods of this section will be used to sample from a multivariate distribution for x = (x1,...,xn) by sampling repeatedly for each variable in turn. To update xi, we must be able to compute a function, fi(xi), that is proportional to p(xi|{xj}j̸=i), where {xj}j̸=i are the values of the other variables.

Yet I still can find no proof of correctness.

Answer 2

It is generally the case that any valid MCMC scheme for univariate distributions can be applied to a univariate conditional distribution as part of an MCMC scheme for sampling from a multivariate distribution. This fact is used extensively all over the literature on MCMC. Its proof is straightforward. There is no need to prove anything special in this regard for slice sampling in particular.