I need to get samples from an unnormalized distribution $p(\theta, \tau | D)$. However, sampling directly from the joint distribution with Metropolis-Hastings is hard, as the sampler rarely finds acceptable states. So, I am trying to do Gibbs sampling by alternatingly sampling from the conditional distributions $p(\theta|\tau, D)$ and $p(\tau|\theta, D)$ with separate Metropolis-Hastings (MH) samplers.
Each time I update $\theta$ or $\tau$, I am changing the conditional distributions $p(\theta|\tau, D)$ and $p(\tau|\theta, D)$. So, if I understand correctly, the MH samplers would need a burn in period to draw reasonable samples from the intended distributions.
My problem is, when I set a small burn in period of just 30 iterations in the MH samplers, each sweep of the Gibbs sampling takes longer than 10 seconds. Running this for merely 4000 iterations would take over half a day.
I have a few related questions that probably belong to the same thread.
I have a separate burn in period for the Gibbs sampler too. In this scenario, is it necessary to have burn in periods for both the Gibbs sampler and the Metropolis-Hastings?
In Metropolis-Hastings, a burn in period of 30 iterations is probably very small. Should I increase it? I am worried that this might take too long.
In this kind of situation, how would you speed up the code? Would you recommend any other algorithm, or any variant of Metropolis-Hastings or Gibbs?