I have some problems understanding the definition of Gibbs sampling.

Let us take into consideration a bivariate distribution \begin{equation} \pi(x_1,x_2): S \subset \mathcal{R^2} \rightarrow \mathcal{R} \end{equation}

If I understand correctly, in order to implement the Gibbs sampling on $\pi(x_1,x_2)$, I should be able to have the full analytical expression of the conditionals with respect to $x_1$ and $x_2$. Let us call them $\pi(x_1|x_2)$ and $\pi(x_2|x_1)$ respectively.

Once I have these expressions, I can sample directly -- i.e. numerically -- from them, one at the time, in the way it is stated by the Gibbs algorithm.

There might be cases in which, the normalization for the conditional distributions might not be so easy to calculate. In this case, I would be driven to perform a Metropolis-Hastings sampling instead of direct sampling on the conditionals.

In the case of direct sampling, at each step $k$ I would extract a random value -- for instance $\tilde{x_2}$ -- for a conditional. I would then accept it as the $k$-th value of my chain $x_2^{(k)}$ with regards to its normalized probability $\pi(\tilde{x}_2|x_1^{(k-1)})$.

When the normalization factor is hard to calculate, my idea would be that of proposing a "jump" from the current $(k-1)$th value of $x_2$ to the randomly extracted value $\tilde{x}_2$. I would then accept it with the Metropolis-Hastings acceptance ratio for the appropriate conditional. In the case of a simmetric proposal distribution this ratio would for example be $\alpha = \frac{\pi(\tilde{x}_2|x_1^{(k-1)})}{\pi(x_2^{k-1}|x_1^{(k-1)})}$.

If I were to sample on the conditionals in this way, would the resulting algorithm still be considered an implementation of the Gibbs algorithm?


1 Answer 1


The main point of Gibbs sampling is that the full conditionals should be available, not the marginals. By the Hammersley-Clifford theorem, these full conditionals contain the same information as the joint distribution. The marginals do not.

In the case the full conditionals are not (all) standard a Metropolis within Gibbs solution (this may be the keyword missing for the OP) is to target the non-standard full conditionals by one single Metropolis step when simulating directly is not feasible in a reasonable time/computational effort.

The missing normalisation terms in the full conditionals are however not an issue, since most simulations techniques including Metropolis do not require the associated normalising constants.

  • 1
    $\begingroup$ You are right, I expressed myself badly. I am editig the question. Still, what if cannot calculate their normalization? $\endgroup$
    – 0x90
    Apr 18, 2019 at 10:44
  • $\begingroup$ Normalisation constants do not matter for the Metropolis step, even when those constants depend on the conditioning variates. $\endgroup$
    – Xi'an
    Oct 10, 2023 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.