Suppose that $y_1, y_2$ are data drawn from a density function $f$ with parameters $\theta_1, \theta_2$ which are unknown. Suppose I applied some prior on $\theta_1, \theta_2$. Then, the Gibbs sampler states that if we know the FULL conditional distributions:

$$ p(\theta_1|\theta_2, y) \ \ \ \text{and} \ \ \ p(\theta_2|\theta_1, y) $$

then we may draw from each and be able to approximate draws from the joint distribution $p(\theta_1, \theta_2)$, which is assumed to be hard to find (which is why we do Gibbs here to begin with).

We may rewrite the above as:

$$ p(\theta_1|\theta_2, y) = \frac{p(\theta_1,\theta_2, y)}{p(\theta_2, y)} \ \ \ \text{and} \ \ \ p(\theta_2|\theta_1, y) = \frac{p(\theta_2,\theta_1, y)}{p(\theta_1, y)} $$

Now, I assume that we know $p(\theta_1, y)$ and $p(\theta_2, y)$ from the prior specification. Then, if we know $p(\theta_1, y)$ and $p(\theta_1|\theta_2, y)$, don't we also know $p(\theta_1,\theta_2, y)$?

What exactly is the Gibb's sampler doing here?


Mathematically, this is correct: once you know $$p(\theta_1|\theta_2, y) \ \ \ \text{and} \ \ \ p(\theta_2|\theta_1, y)$$ you can derive the joint posterior distribution as $$p(\theta_1,\theta_2|y) = \dfrac{p(\theta_1|\theta_2,y)}{\int p(\theta_1|\theta_2,y)\big/p(\theta_2|\theta_1, y)\,\text{d}\theta_1}$$ [This is Theorem 9.3 in our book.]

However the reason for running Gibbs sampling is that/when this expression is not available in closed form and hence cannot be simulated directly. In the event one knows $p(\theta_1,y)$ or $p(\theta_2,y)$ in closed form and can simulate $\theta_1$ or $\theta_2$ from the former or the latter, resp., then one does not require Gibbs sampling as direct simulation becomes feasible.

  • $\begingroup$ So you're basically saying that even if I can compute $p(\theta_1,\theta_2, y)$ from the full conditionals, to get to $p(\theta_1,\theta_2|y)$, I still need the $p(y)$ on the denominator, which is the normalizing constant and hence whose closed form is the deciding factor in whether to use Gibbs or not? $\endgroup$ – user321627 Feb 16 '17 at 6:45
  • $\begingroup$ I see, so $\int p(\theta_1|\theta_2,y)\big/p(\theta_2|\theta_1, y)\,\text{d}\theta_1 = \frac{1}{p(\theta_2,y)}\int p(\theta_1, y)d\theta_1 = \frac{p(y)}{p(\theta_2,y)}$. For the closed form and simulation requirement, is to facilitate the calculation of the integral part $\int p(\theta_1, y)d\theta_1$ or the fraction part $\frac{1}{p(\theta_2,y)}$? Thank you! $\endgroup$ – user321627 Feb 16 '17 at 9:23
  • 1
    $\begingroup$ Gibbs sampling does not aim at recovering the joint density, only at simulating from this density. $\endgroup$ – Xi'an Feb 16 '17 at 11:21
  • $\begingroup$ When you say "However the reason for running Gibbs sampling is that/when this expression is not available in closed form and hence cannot be simulated directly.", when is it NOT available in closed form? Is it when you cannot evaluate the integral in the denominator above? $\endgroup$ – user321627 Feb 24 '17 at 22:12
  • $\begingroup$ @user321627: when the integral is not available in closed form or when it involves an expression complicated enough to prevent simulating from this distribution, conditionals need be used instead. $\endgroup$ – Xi'an Apr 1 '18 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.