2
$\begingroup$

I need to get samples from an unnormalized distribution $p(\theta, \tau | D)$. However, sampling directly from the joint distribution with Metropolis-Hastings is hard, as the sampler rarely finds acceptable states. So, I am trying to do Gibbs sampling by alternatingly sampling from the conditional distributions $p(\theta|\tau, D)$ and $p(\tau|\theta, D)$ with separate Metropolis-Hastings (MH) samplers.

Each time I update $\theta$ or $\tau$, I am changing the conditional distributions $p(\theta|\tau, D)$ and $p(\tau|\theta, D)$. So, if I understand correctly, the MH samplers would need a burn in period to draw reasonable samples from the intended distributions.

My problem is, when I set a small burn in period of just 30 iterations in the MH samplers, each sweep of the Gibbs sampling takes longer than 10 seconds. Running this for merely 4000 iterations would take over half a day.

I have a few related questions that probably belong to the same thread.

Questions:

  1. I have a separate burn in period for the Gibbs sampler too. In this scenario, is it necessary to have burn in periods for both the Gibbs sampler and the Metropolis-Hastings?

  2. In Metropolis-Hastings, a burn in period of 30 iterations is probably very small. Should I increase it? I am worried that this might take too long.

  3. In this kind of situation, how would you speed up the code? Would you recommend any other algorithm, or any variant of Metropolis-Hastings or Gibbs?

$\endgroup$
2
$\begingroup$

This is a common misunderstanding about Metropolis-within-Gibbs: at each Gibbs step, the Metropolis-within-Gibbs need not converge! A single iteration is enough. This means running $T$ iterations of the Gibbs sampler, i.e., $2\times T$ simulations of the full conditionals $p(\theta|\tau,D)$ and $p(\tau|\theta,D)$. And equally $2\times T$ Metropolis calls to replace exact simulations from $p(\theta|\tau,D)$ and $p(\tau|\theta,D)$. Concerning the burnin for the Gibbs sampler, it will depend on the correlation between $\theta$ and $\tau$ so it is unwise to advise a fixed number out of the blue.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.