I heard that all the full conditionals (as used in Gibbs sampling) can determine the joint distribution. But I don't understand why and how. Or did I mis-hear? Thanks!
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
This seemingly simple question is deeper than it looks, leading us all the way to the Hammersley-Clifford theorem. The fact that we can recover the joint distribution from the full conditionals is what makes the Gibbs sampler possible. It may be seen as a surprising result, if we remember that the marginals do not determine the joint distribution.
Let's see what happens if we compute formally with the well-known definitions of the joint, conditionals and marginals densities. Since $$ f_{X,Y}(x,y)=f_{X\mid Y}(x\mid y)\,f_Y(y)=f_{Y\mid X}(y\mid x)\,f_X(x) \, , $$ we have $$ \int \frac{f_{Y\mid X}(y\mid x)}{f_{X\mid Y}(x\mid y)}dy = \int \frac{f_Y(y)}{f_X(x)}dy = \frac{1}{f_X(x)} \, , $$ and we can formally recover the joint density from the full conditionals making $$ f_{X,Y}(x,y) = \frac{f_{Y\mid X}(y\mid x)}{\int f_{Y\mid X}(y\mid x)/f_{X\mid Y}(x\mid y)\,dy} \, . \qquad (*) $$
The problem with this formal computation is that it supposes that all the involved objects do exist.
For instance, consider what happens if we are given that $$ X\mid Y=y\sim\text{Exp}(y) \qquad \text{and} \qquad Y\mid X=x\sim\text{Exp}(x) \, . $$ It follows that $f_{Y\mid X}(y\mid x)/f_{X\mid Y}(x\mid y) = x /y$, and the integral in the denominator of $(*)$ diverges.
To guarantee that we can recover the joint density from the full conditionals using $(*)$ we need the compatibility conditions discussed in this paper:
"Compatible Conditional Distributions", Barry C. Arnold and S. James Press, Journal of the American Statistical Association, Vol. 84, No. 405 (1989), pp. 152-156.
Finally, read the discussion on the Hammersley-Clifford Theorem in Robert and Casella's book
-
1$\begingroup$ Could you clarify what is meant by "the integral .... exists"? There seem to be two different issues here, viz. (i) does the integral $$\int \frac{f_{Y\mid X}(y\mid x)}{f_{X\mid Y}(x\mid y)}dy$$ exist or not? and (ii) if the integral exists, is its value $\frac{1}{f_X(x)}$? Or are you saying that whenever $X$ and $Y$ have conditional densities such that $$\int \frac{f_{Y\mid X}(y\mid x)}{f_{X\mid Y}(x\mid y)}dy$$ exists, then it must be that the value of the integral is $\frac{1}{f_X(x)}$? $\endgroup$ Mar 5, 2014 at 23:40
-
$\begingroup$ Thanks, @Zen! $f_Y$ and $f_{X|Y}$ can determine $f_{X,Y}$, and $f_{Y|X}$ and $f_{X|Y}$ can also determine $f_{X,Y}$. (1) Which one provides more information, $f_Y$ or $f_{Y|X}$? (2) Which one provides less redundant/overlapping information with $f_{X|Y}$, $f_Y$ or $f_{Y|X}$? (3) Out of $f_Y$ and $f_{Y|X}$, does one of them already provide the information of the other (which I doubt, because that would imply one leads to the other)? I guess it is the "intersection" between the info of $f_Y$ and of $f_{Y|X}$, which together with $f_{X|Y}$ determines $f_{X,Y}$. $\endgroup$– TimMar 6, 2014 at 13:35
-
$\begingroup$ Hi @Tim. $f_Y$ represents you uncertainty about $Y$, while $f_{Y\mid X}$ represents your uncertainty about $Y$, given that you know the value of $X$. "Which one contains more information?" is not an easy question. If $f_{X\mid Y}$ and $f_{Y\mid X}$ are compatible (in the sense of Arnold and Press), then they determine $f_{X,Y}$ through $(*)$. $\endgroup$– ZenMar 6, 2014 at 16:43
-
$\begingroup$ I'm currently struggeling with the same problem. I'm a bit confused by the need for compatible conditional distributions, as these are never mentioned in any (at least the ones I have read) introductions to Gibbs Sampling. Or does the need for compatible conditional distributions only hold, if one tries to formally recover the joint distributions, e. g. by (*). -> not approximating the joint distribution by Gibbs Sampling? $\endgroup$– sklingelNov 30, 2015 at 12:17
-
$\begingroup$ In a regular Gibbs sampling setting applied to a statistical problem you assume that the joint probability (posterior) distribution exists, hence that the full conditionals derived from this joint distribution are compatible. Outside this case, Gibbs sampling is meaningless. $\endgroup$– Xi'anNov 30, 2015 at 12:50