Typically in Gibbs sampling we want to sample from a joint distribution $p(X_1, X_2, ..., X_N)$, but because the joint is hard to sample from directly, we instead achieve this by iteratively sampling each variable $X_i$ from its conditional distribution $p(X_i|\{X_{-i}\})$. The resulting samples then allow us to approximate the full joint distribution of $\{X_i\}$, or any of the marginals $p(X_i)$. If we cannot sample directly from the conditionals, we can insert a Metropolis-Hastings (MH) step, which results in the Metropolis-Within-Gibbs algorithm.

The problem I have is one in which, for a subset of the $X_i$'s, I actually can't evaluate their conditional distribution, but I can easily sample from their (joint) marginal distribution. To simplify this a bit, suppose I want to sample from the joint distribution $p(X_1, X_2)$, and I only have access to the marginal distribution $p(X_1)$ and the conditional $p(X_2|X_1)$. Additionally, I cannot sample from the conditional directly, so I have to use a MH step there.

One simple algorithm that works is to do the following:

1. Sample $X_1^*$ from $p(X_1)$
2. Given this sample $X_1^*$, generate a MH chain of samples $X_2^{(t)*}|X_1^*,X_2^{(t-1)*}$

If the chain in step 2 is run for sufficient length, it will converge to the target distribution $p(X_2|X_1^*)$, and so the final sample of $X_2$ will be a proper sample from the conditional. However, therein lies my problem, because I'm working in much higher dimensions than this, and so it would take a long time for this chain to converge, and I can't afford to run a long MCMC chain just to get a single sample of my variables.

Is there any solution to this, e.g. a way to use short chains in step 2 with some sort of correction?

Typically in Gibbs sampling we want to sample from a joint distribution $p(X_1, X_2, ..., X_N)$, but because the joint is hard to sample from directly, we instead achieve this by iteratively sampling each variable $X_i$ from its conditional distribution $p(X_i|\{X_{-i}\})$. The resulting samples then allow us to approximate the full joint distribution of $\{X_i\}$, or any of the marginals $p(X_i)$. If we cannot sample directly from the conditionals, we can insert a Metropolis-Hastings (MH) step, which results in the Metropolis-Within-Gibbs algorithm.

The problem I have is one in which, for a subset of the $X_i$'s, I actually can't evaluate their conditional distribution, but I can easily sample from their (joint) marginal distribution. To simplify this a bit, suppose I want to sample from the joint distribution $p(X_1, X_2)$, and I only have access to the marginal distribution $p(X_1)$ (more specifically, I can sample from this distribution, but I cannot evaluate it) and the conditional $p(X_2|X_1)$. Additionally, I cannot sample from the conditional directly, so I have to use a MH step there.

One simple algorithm that works is to do the following:

1. Sample $X_1^*$ from $p(X_1)$
2. Given this sample $X_1^*$, generate a MH chain of samples $X_2^{(t)*}|X_1^*,X_2^{(t-1)*}$

If the chain in step 2 is run for sufficient length, it will converge to the target distribution $p(X_2|X_1^*)$, and so the final sample of $X_2$ will be a proper sample from the conditional. However, therein lies my problem, because I'm working in much higher dimensions than this, and so it would take a long time for this chain to converge, and I can't afford to run a long MCMC chain just to get a single sample of my variables.

Is there any solution to this, e.g. a way to use short chains in step 2 with some sort of correction?

Typically in Gibbs sampling we want to sample from a joint distribution $p(X_1, X_2, ..., X_N)$, but because the joint is hard to sample from directly, we instead achieve this by iteratively sampling each variable $X_i$ from its conditional distribution $p(X_i|\{X_{-i}\})$. The resulting samples then allow us to approximate the full joint distribution of $\{X_i\}$, or any of the marginals $p(X_i)$. If we cannot sample directly from the conditionals, we can insert a Metropolis-Hastings (MH) step, which results in the Metropolis-Within-Gibbs algorithm.

The problem I have is one in which, for a subset of the $X_i$'s, I actually can't evaluate their conditional distribution, but I can easily sample from their (joint) marginal distribution. To simplify this a bit, suppose I want to sample from the joint distribution $p(X_1, X_2)$, and I only have access to the marginal distribution $p(X_1)$ and the conditional $p(X_2|X_1)$. Additionally, I cannot sample from the conditional directly, so I have to use a MH step there.

One simple algorithm that works is to do the following:

1. Sample $X_1^*$ from $p(X_1)$
2. Given this sample $X_1^*$, generate a MH chain of samples $X_2^{(t)*}|X_1^*,X_2^{(t-1)*}$

If the chain in step 2 is run for sufficient length, it will converge to the target distribution $p(X_2|X_1)$, and so the final sample of $X_2$ will be a proper sample from the conditional. However, therein lies my problem, because I'm working in much higher dimensions than this, and so it would take a long time for this chain to converge, and I can't afford to run a long MCMC chain just to get a single sample of my variables.

Is there any solution to this, e.g. a way to use short chains in step 2 with some sort of correction?

Typically in Gibbs sampling we want to sample from a joint distribution $p(X_1, X_2, ..., X_N)$, but because the joint is hard to sample from directly, we instead achieve this by iteratively sampling each variable $X_i$ from its conditional distribution $p(X_i|\{X_{-i}\})$. The resulting samples then allow us to approximate the full joint distribution of $\{X_i\}$, or any of the marginals $p(X_i)$. If we cannot sample directly from the conditionals, we can insert a Metropolis-Hastings (MH) step, which results in the Metropolis-Within-Gibbs algorithm.

The problem I have is one in which, for a subset of the $X_i$'s, I actually can't evaluate their conditional distribution, but I can easily sample from their (joint) marginal distribution. To simplify this a bit, suppose I want to sample from the joint distribution $p(X_1, X_2)$, and I only have access to the marginal distribution $p(X_1)$ and the conditional $p(X_2|X_1)$. Additionally, I cannot sample from the conditional directly, so I have to use a MH step there.

One simple algorithm that works is to do the following:

1. Sample $X_1^*$ from $p(X_1)$
2. Given this sample $X_1^*$, generate a MH chain of samples $X_2^{(t)*}|X_1^*,X_2^{(t-1)*}$

If the chain in step 2 is run for sufficient length, it will converge to the target distribution $p(X_2|X_1^*)$, and so the final sample of $X_2$ will be a proper sample from the conditional. However, therein lies my problem, because I'm working in much higher dimensions than this, and so it would take a long time for this chain to converge, and I can't afford to run a long MCMC chain just to get a single sample of my variables.

Is there any solution to this, e.g. a way to use short chains in step 2 with some sort of correction?