Suppose I want to sample from the joint distribution $p(X, Y)$, where $X$ is a random variable and $Y = f(X)$ where $f$ is a known function of $X$. However, sampling from $p(X,Y)$ directly is hard. Could I use Gibbs sampling and sample from the conditionals $x^{(k)} \sim p(X\mid Y=y^{(k-1)})$ with $y^{(k-1)}=f(x^{(k-1)})$ and $y^{(k)} \sim p(Y\mid X=x^{(k)}) = \delta(y-f(x^{(k)}))$ for $k = 1, 2, \dots$? Would this Gibbs sampler converge?
$\begingroup$
$\endgroup$
9
-
2$\begingroup$ Sampling from $p(X,Y)$ directly is easy: sample $X$ and compute $Y=f(X)$. $\endgroup$– whuber ♦Feb 10, 2017 at 18:49
-
$\begingroup$ This would work if I could decompose $f(X, Y) = f(Y|X)f(X)$. But I do not know the marginal $f(X)$. $\endgroup$– AnselmoFeb 10, 2017 at 18:59
-
1$\begingroup$ I do not follow that at all. If you can draw values from $X$, then you can draw values from $f(X)$ by applying $f$ to them. According to your description, the recipe I gave is practically the only way to draw values from $(X,Y)$, and it's computationally optimal, because any method to draw from $(X,Y)$ in any manner yields draws from $X$ (by ignoring $Y$) and having the $Y$-values amounts to computing $f(X)$ for each draw. Note that the joint distribution is singular: it does not have a density. $\endgroup$– whuber ♦Feb 10, 2017 at 19:07
-
$\begingroup$ Sorry, I meant the marginal $p(X) = \int p(X, Y)dY$, not the the function $f(X)$. I know how to compute $f(X)$ given $X$. But I do not have the marginal $p(X)$. $\endgroup$– AnselmoFeb 10, 2017 at 19:12
-
$\begingroup$ I made a confusion with letters $p$ and $f$. $\endgroup$– AnselmoFeb 10, 2017 at 19:13
|
Show 4 more comments
1 Answer
$\begingroup$
$\endgroup$
1
(This answer does not really help with the Gibbs sampler much, but points at something else you could do. I do think the Gibbs sampler would converge).
Recall that, $$p(X,Y) = p(Y | X) p(X)\, . $$
Now if you can sample from the marginal of $X$, then $X$ is the linchpin variable. You can use exact sampling methods if it is a known distribution, or you could use MCMC. This sampler should intuitively converge faster than the Gibbs sampler because the Markov chain is only present in $X$ here, whereas in the Gibbs sampler, the Markov chain samples both $(X, Y)$.
answered Feb 10, 2017 at 18:50
-
$\begingroup$ Thanks for the reference. The problem is that I do not know the marginal $p(X)$. I know the conditional $p(X|Y)$, i.e., for fixed $Y = y$, I know how to sample $X$. And for fixed $X = x$, I can compute $Y = f(X)$. $\endgroup$– AnselmoFeb 10, 2017 at 19:16