Confusion in Gibbs sampling

Question

I am self-studying Gibbs sampling from a book. The book introduces metropolis hastings algortihm to generate representative values from a posterior distribution. So we know $p(D | \theta) p(\theta)$ but not in the normalized form. So we generate those values.

Now when it introduces Gibbs sampling it quotes,

Gibbs sampling is especially useful when the complete joint posterior, $p(θ_i |D)$, cannot be analytically determined and cannot be directly sampled, but all the conditional distributions, $ p(θ_i |\{θ_{j \ne i}\}, D)$, can be determined and directly sampled.

Suppose we have two parameters $\theta_1$ and $\theta_2$ so for determining $p(θ_1|θ_2, D)$, we need $\frac{p(θ_1, θ_2|D)}{p(θ_2|D)}$, and now I am all the more confused. First it says

when we can't analytically determine the posterior

and then uses $\frac{p(θ_1, θ_2|D)}{p(θ_2|D)}$ , where we don't know yet the posterior distribution, because that is not "analytically determined". Can anyone explain, what am I understanding wrong.

Answer 1

Since I'm not sure where are you stuck at, I'll try multiple shots:

Explanation 1: The thing is that you only need the form of the unnormalized posterior, and that is why it's enough if you can get: $$ p(\theta_1 | \theta_2, D) \propto p(\theta_1, \theta_2 | D) $$ The normalizing constant is not interesting, this is very common in bayesian statistics, . With Gibbs sampling, Metropolis-Hasting or any other Monte Carlo method, what you are doing is drawing samples from this posterior. That is, the more density around a point, the more samples you'll get there.

Then, once you have enough samples from this posterior distribution, you know that the normalized density at some point $x$ is the proportion of samples that felt at that point.

You can even plot an histogram on the samples to see the (unnormalized) posterior.

In other words, if I give you the samples $1,3,4,5,1.....,3,4,16,1$ and I tell you these are samples from a density function, you know to compute the probability of every value.

Explanation 2: If you observe the analytical form of your unnormalized posterior (you always know it [1]), two things can happen:

a) It has the shape of some known distribution (e.g. Gaussian): then you can get the normalize posterior since you know the normalizing constant of a gaussian distribution.

b) It has an ugly form that corresponds to no familiar distribution: then you can always sample with Metropolis-Hastings (there are others).

b.1) M-H is not the most efficient of the methods (you reject a lot of samples, usually more than 2/3). If the posterior was ugly but the conditionals of the individual variables are pretty (known distributions) then you can do Gibbs sampling by sampling for one single variable at a time.

Explanation 3 If you use conjugate priors for the individual variables, the denominator of their conditional probability will be always nice and familiar, and you will know what the normalizing constant in the denominator is. This is why Gibbs sampling is so popular when the joint probability is ugly but the conditional probabilities are nice.

Maybe this thread, and specially the answer with puppies, helps you: Why Normalizing Factor is Required in Bayes Theorem?

[1] Edit: not true, see @Xi'an's comment.

Update (example)

Imagine you have:

$$ P(\theta_1, \theta_2 | D) = \frac{ p(D , \theta_1, \theta_2)} {\int_{\theta_1, \theta_2} p(D , \theta_1, \theta_2) \text{d} \theta_1, \theta_2} \propto p(D , \theta_1, \theta_2) $$

If the joint probability is complicated, then you can't know the normalization constant. Sometimes, if it does not contain things like large $\sum$ or $\prod$ that would make it painful to compute, you can even plot the posterior. In this case you would have some 2-D plot with axes $\theta_1$ and $\theta_2$. Yet, you plot is right up to a missing constant. Sampling algorithms say "ok, I don't know what the normalization factor is, but if I might draw samples from this function in such a way that, if $p(D, \theta_1=x_1, \theta_2=x_2)$ is two times $p(D, \theta_1=x_3, \theta_2=x_4)$, then then I should get the sample $(x_1, x_2)$ twice as much as $(x_3, x_4)$"

Gibbs sampling does this by sampling every variable separatedly. Imagine $\theta_1$ is a mean $\mu$ and that its conditional probability is (forget about the $\sigma$'s, imagine we know them): $$ p(\mu | D) = \frac{ \mathcal{N}(D | \mu, \sigma_d) \mathcal{N}(\mu, \sigma) } {\int \mathcal{N}(D | \mu, \sigma_d) \mathcal{N}(\mu, \sigma) \text{d} \mu} $$ The product of two normals is another normal with new parameters (see conjugate priors and keep that table always at hand. Even memorize the ones you end up using the most). You do the multiplication, you drop everything that does not depend on $\mu$ into a constant $K$ and you get something that you can express as: $$ p(\mu | D) = K \exp\left(\frac{1}{a}(\mu - b)^2\right) $$ It has the functional form of a Gaussian. Therefore since you know it is a density, $K$ must be the normalizing factor of $\mathcal{N}(b, a)$.

Thus, your posterior is a Gaussian distribution with posterior parameters (b,a).

The short version is that the product of the prior and the likelihood have the functional form of a familiar distribution (it actually has the form of the prior if you chose conjugates), then you know how to integrate it. For instance, the integral of the $\exp(...)$ element of a normal distribution, that is, a normal without its normalizing factor, is the inverse of its normalizing factor.