If the Gibbs Sampler assumes knowledge of full conditionals — why can't we just solve for the full joint density and avoid Gibbs?

Question

Suppose that $y_1, y_2$ are data drawn from a density function $f$ with parameters $\theta_1, \theta_2$ which are unknown. Suppose I applied some prior on $\theta_1, \theta_2$. Then, the Gibbs sampler states that if we know the FULL conditional distributions:

$$ p(\theta_1|\theta_2, y) \ \ \ \text{and} \ \ \ p(\theta_2|\theta_1, y) $$

then we may draw from each and be able to approximate draws from the joint distribution $p(\theta_1, \theta_2)$, which is assumed to be hard to find (which is why we do Gibbs here to begin with).

We may rewrite the above as:

$$ p(\theta_1|\theta_2, y) = \frac{p(\theta_1,\theta_2, y)}{p(\theta_2, y)} \ \ \ \text{and} \ \ \ p(\theta_2|\theta_1, y) = \frac{p(\theta_2,\theta_1, y)}{p(\theta_1, y)} $$

Now, I assume that we know $p(\theta_1, y)$ and $p(\theta_2, y)$ from the prior specification. Then, if we know $p(\theta_1, y)$ and $p(\theta_1|\theta_2, y)$, don't we also know $p(\theta_1,\theta_2, y)$?

What exactly is the Gibb's sampler doing here?

Answer 1

Mathematically, this is correct: once you know $$p(\theta_1|\theta_2, y) \ \ \ \text{and} \ \ \ p(\theta_2|\theta_1, y)$$ you can derive the joint posterior distribution as $$p(\theta_1,\theta_2|y) = \dfrac{p(\theta_1|\theta_2,y)}{\int p(\theta_1|\theta_2,y)\big/p(\theta_2|\theta_1, y)\,\text{d}\theta_1}$$ [This is Theorem 9.3 in our book.]

However the reason for running Gibbs sampling is that/when this expression is not available in closed form and hence cannot be simulated directly. In the event one knows $p(\theta_1,y)$ or $p(\theta_2,y)$ in closed form and can simulate $\theta_1$ or $\theta_2$ from the former or the latter, resp., then one does not require Gibbs sampling as direct simulation becomes feasible.