5
$\begingroup$

I read on this thread the following:

If you can use both the gibbs sampler and slice sampling to sample from a posterior I would use the Gibbs sampler as the slice sampler seems unnecessary to me <...> If you cannot use the gibbs but you can use the slice then your question seems irrelevant.

But then, when on these slides from M. Jordan's lecture he says that:

Slice sampling is a special case of Gibbs sampling (in a product space).

What does he mean by that?

Also, when would one not be able to do Gibbs sampling but still do slice sampling?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

Slice sampling is a special case of Gibbs sampling, where one introduces a latent uniform random variable and then applies Gibbs to the extended set of rvs - this is what Jordan means by in a product space. Slice sampling can also be used within an existing Gibbs scheme, again by augmenting the parameter space (although one does not need to store the uniform parameters during sampling) so I guess what the answer in the other thread means is that it makes no sense to augment an existing Gibbs sampler with slice-within-Gibbs (EDIT: unless you can't run the existing Gibbs in the first place, of course).

It's worth pointing out that there are generalizations of univariate slice sampling (e.g. reflective or elliptical) that can perform much better than Gibbs sampling, so I don't agree with the other answerer about never preferring slicing over Gibbs.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks. Would you mind clarifying when would one not be able to run Gibbs in first place, but still be able to run slice-sampling? $\endgroup$
    – Amelio Vazquez-Reina
    Aug 13, 2014 at 23:19
  • 1
    $\begingroup$ @user023472 Suppose we have parameters $(\theta_1, \theta_2)$ and data $Y$. Gibbs sampling iterates between $p(\theta_1 \mid \theta_2, y)$ and $p(\theta_2 \mid \theta_1, y)$; if one cannot sample exactly from either of these two distributions, Gibbs sampling will not be feasible. But we can always apply the slice sampler, so we can replace the draws from the intractable distributions with draws from slice samplers which target to intractable distributions. $\endgroup$
    – guy
    Aug 13, 2014 at 23:35
  • $\begingroup$ @guy I really want to use slice sampling that way, but I can't find a reference to justify it. Do you know of one? $\endgroup$
    – landau
    Jan 27, 2016 at 1:28
  • $\begingroup$ @landau you want to use it in what way? $\endgroup$
    – guy
    Jan 27, 2016 at 4:16
  • 2
    $\begingroup$ @landau Yes, there is theory to support it. It is easy to show that any transition kernel which leaves a given full conditional invariant will also leave the joint distribution invariant. The only other condition required for MCMC to work is that the chain is irreducible, which means that the chain can get to any state (plus some other technicalities). This idea is what is powering JAGS, which uses slice sampling to sample the conditionals almost exclusively. I don't have an explicit source for this idea, though, because it is so well known. $\endgroup$
    – guy
    Jan 27, 2016 at 15:09
2
$\begingroup$

I disagree that slice sampling is strictly a special case of Gibbs sampling. It's almost Gibbs sampling, but not quite. Look at Neal (2003) section 3, which talks about sampling from $f(x)$ using an auxiliary variable $y$. A Gibbs sampler would

  1. Sample $y$ from $p(y | x)$ = Uniform(0, $f(x)$).
  2. Sample $x$ from $p(x | y)$ = Uniform($S$), where $S = \{x: y < f(x)\}$.

Step 1 is an exact Gibbs step, and if we knew the slice $S$ exactly, step 2 could be as well. However, we don't know $S$ in practice, so slice sampling uses successive approximations of $S$ instead of $S$ itself. Different ways to do this included the stepping-out slice sampler and the doubling slice sampler, both suggested by Neal. But in any case, step 2 is not an exact, true Gibbs step.

I've read that slice sampling is a special case of a general auxiliary variable method, which relies on the same data augmentation idea: i.e., introducing an extra variable like $y$ and using it similarly to make sampling $x$ easier.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Slice sampling predates the 2003 paper by Neal. The original algorithm, which is precisely the one you gave here, was given the name "slice sampling" by Neal in 1997, although the idea goes back to the late 80s. The step-out algorithm is really an attempt at mimicking the original slice sampler you gave. So, when someone says that slice sampling is just Gibbs, they mean the original algorithm, not the step-out algorithm Neal came up with as an attempt to mimic the original slice sampler. $\endgroup$
    – guy
    Jan 27, 2016 at 15:22
  • 1
    $\begingroup$ Incidentally, there is an entire chapter of Robert and Casella titled "Slice Sampling" that consists entirely of methods which implement steps 1 and 2 exactly; Neal's step-out algorithm is referred to in the bibliographic material, although it is probably more important than the material in the chapter itself. But this should drive home the point that slice sampling is not synonymous with the step-out algorithm. $\endgroup$
    – guy
    Jan 27, 2016 at 15:25
  • $\begingroup$ I didn't know slice sampling could also refer to the exact Gibbs sampler. $\endgroup$
    – landau
    Jan 27, 2016 at 19:04
  • $\begingroup$ After correcting the terminology, is my original answer still wrong? Is Neal's step-out algorithm a special case of Gibbs sampling too? If so, the connection is not obvious to me. $\endgroup$
    – landau
    Apr 6, 2016 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.