# Approximately sampling $(X, Y)$ when sampling $X$ is easy

Suppose I am interested in sampling many pairs $(\mathbf X, Y)$ from some distribution $f(\mathbf x, y)$ where $\mathbf x \in \mathbb R^p$, $p$ large ; I am interested in both exact and approximate simulations. $f(\mathbf x)$ is easy to sample from, but $f(y \mid \mathbf x)$ is not.

For motivation, I could do Gibbs sampling if the distributions $f(\mathbf x\mid y)$ and $f(y \mid \mathbf x)$ were both easy to simulate from by initializing $(\mathbf X_0, Y_0)$ and drawing $\mathbf X_t \sim f(x \mid Y_{t-1})$ and $Y_t \sim f(y \mid \mathbf X_t)$. If $f(\mathbf x \mid y)$ is easy to sample from, then I am in really good shape because, worst case scenario, I can replace $f(y \mid \mathbf x)$ with any update that leaves this distribution invariant.

In my situation, $f(\mathbf x \mid y)$ is difficult to sample from, and substantial work would have to go into constructing a suitable transition kernel given the dimension of $\mathbf x$. However, I can draw from the marginal $f(\mathbf x)$ exactly! It seems like this should buy me something like it does with the Gibbs sampler, however if I do something like the following:

1. Draw $\mathbf X_t \sim f(\mathbf x)$;
2. Draw $Y_t \sim K(y \mid \mathbf X_t, Y_{t-1})$ where $K(\cdot \mid \mathbf x, y)$ leaves $f(y \mid \mathbf x)$ invariant,

I believe I will not get the correct stationary distribution.

If it helps, I might be willing to evaluate the density $f(y \mid \mathbf x)$. I know $f(y \mid \mathbf x)$ up-to a normalizing constant, but I can afford to do one numerical integration to get the constant (I would really prefer not to, though). However, $f(y \mid \mathbf x)$ is expensive to compute.

One thought I've had is to generate $f(\mathbf x)$ and then do a small number of MH random walk steps, but I would very much prefer getting an answer that will at least converge to the correct thing as $t \to \infty$ (this one will always have some error if I do a small number of steps for each $Y_t$).

EDIT: Maybe this all seems obvious - just use generic sampling techniques for each $f(y \mid \mathbf x)$ and, in general, I shouldn't be able to do better. I guess what I have in mind is that I should somehow be able to get the MCMC to work across $t$, and not have to do my approximate sampling within $t$. That is, I want my approximation to get better and better as $t \to \infty$; I don't want my approximation to have the same amount of error within $t$.

In general, though, $f(y \mid \mathbf x)$ might be very different for different values of $\mathbf x$ so I shouldn't hope for a solution that will always work. I just want something that has a good chance of working and addresses the above issues. It is strange to me that $f(\mathbf x)$ would be less useful than $f(\mathbf x \mid y)$.

• If you want something other than a generic, approximate solution then I think you need to provide more details on the exact form of $f(x)$, $f(y)$ and $f(x,y)$. – M. Berk Oct 22 '13 at 8:17