Can full conditionals determine the joint distribution? I heard that all the full conditionals (as used in Gibbs sampling) can determine the joint distribution. But I don't understand why and how. Or did I mis-hear? Thanks!
 A: This seemingly simple question is deeper than it looks, leading us all the way to the Hammersley-Clifford theorem. The fact that we can recover the joint distribution from the full conditionals is what makes the Gibbs sampler possible. It may be seen as a surprising result, if we remember that the marginals do not determine the joint distribution.
Let's see what happens if we compute formally with the well-known definitions of the joint, conditionals and marginals densities. Since
$$
  f_{X,Y}(x,y)=f_{X\mid Y}(x\mid y)\,f_Y(y)=f_{Y\mid X}(y\mid x)\,f_X(x) \, ,
$$
we have
$$
  \int \frac{f_{Y\mid X}(y\mid x)}{f_{X\mid Y}(x\mid y)}dy = \int \frac{f_Y(y)}{f_X(x)}dy = \frac{1}{f_X(x)} \, ,
$$
and we can formally recover the joint density from the full conditionals making
$$
  f_{X,Y}(x,y) = \frac{f_{Y\mid X}(y\mid x)}{\int f_{Y\mid X}(y\mid x)/f_{X\mid Y}(x\mid y)\,dy} \, . \qquad (*)
$$
The problem with this formal computation is that it supposes that all the involved objects do exist. 
For instance, consider what happens if we are given that
$$
  X\mid Y=y\sim\text{Exp}(y) \qquad \text{and} \qquad Y\mid X=x\sim\text{Exp}(x) \, .
$$
It follows that $f_{Y\mid X}(y\mid x)/f_{X\mid Y}(x\mid y) = x /y$, and the integral in the denominator of $(*)$ diverges.
To guarantee that we can recover the joint density from the full conditionals using $(*)$ we need the compatibility conditions discussed in this paper:
"Compatible Conditional Distributions", Barry C. Arnold and S. James Press, Journal of the American Statistical Association, Vol. 84, No. 405 (1989), pp. 152-156.
Finally, read the discussion on the Hammersley-Clifford Theorem in Robert and Casella's book
