As the title suggest, does gibbs sampling require to know the partition function? For example, if I want to sample variable $a$ and I have worked out $p(a|rest) \propto f(a|rest)$ where $rest$ represents the rest of the variables, do I need to work out the partition function $Z$ that satisfies $p(a|rest) = \frac{1}{Z} f(a|rest)$?

  • 1
    $\begingroup$ If you recognize $f$ as the kernel of a density from which you can directly sample, you'd not need to write $p$ down; although "recognizing the kernel" means that you know already the partition function. If you don't know the partition function and don't have a way to directly sample from $p \propto f$ , you could take a slice sampling, rejection sampling, or Mertropolis-Hasting step or any other sampling step that requires only that the target is known up to a constant. $\endgroup$ – baruuum Jul 6 at 6:24

Gibbs Sampling doesn't explicitly require you to know $Z$. It requires you to sample from the full conditionals, i.e. $\frac{1}{Z}f(\alpha|\text{rest})$, in some way which means knowing $Z$ already, or approximating it via various MCMC methods (e.g. Metropolis-Hastings, choosing suitable proposal function etc.) as @baruuum pointed out. So, if you imagine the case that you have a black-box doing the sampling step for you given the target function, (i.e. you just call sample method); Gibbs depends on it, but doesn't need $Z$ to be known since the sampling step is actually doing the integration for you implicitly.

  • $\begingroup$ In another case, say we have two variables $a$ and $b$ that we want to sample, we have $p(a|b,rest) = \frac{1}{Z_a}f(a|b,rest)$ and $p(b|a,rest) = \frac{1}{Z_b}g(b|rest)$, we can work out $f(a|b,rest)$, $g(b|a,rest)$ and $Z_b$ but we do not know $Z_a$, in this case, can we jointly use Gibbs Sampling to sample $b$ and reject sampling to sample $a$? Does the order matter (sample $b$ first or $a$)? $\endgroup$ – Yi Yang Jul 6 at 23:18
  • $\begingroup$ Yes, you can use; and Gibbs doesn’t force any ordering in the sampling scheme. $\endgroup$ – gunes Jul 7 at 7:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.