# How should I implement an adaptive Metropolis algorithm for a Gibbs sampler with two Metropolis steps?

Consider the Gibbs sampler

1. Sample $$\theta' \sim p(\theta|\tau, D)$$
2. Sample $$\tau' \sim p(\tau|\theta', D)$$

Both conditional distributions are sampled with a Metropolis step. The joint distribution is unknown so that only a Gibbs sampler with two separate Metropolis steps can be used.

There are many forms of adaptive Metropolis algorithms which change the covariance matrix $$\Sigma$$ of the proposal (jump-) distribution assumed to be MV-normal and centered at the current values: $$N(x|x',\Sigma)$$. For example, in this question I wrote down the adaptive MH algroithm of Haario et al (1999). In an extension Haario et al. (2001) suggested to evaluate the empirical covariance matrix $$S$$ over the first $$t_0$$ MCMC draws and scaled by a factor $$s=2.4^2/d$$ as proposal variance $$\Sigma$$ (with d the dimension). A similar strategy is suggested by Gelman et al in 'Bayesian Data Analysis'(3rd ed., p. 296).

Now, when there is only one posterior to sample from, the approach is clear. However what should one do in the case of the two MH steps within Gibbs? We then have two proposals $$\Sigma_{\theta}$$ and $$\Sigma_{\tau}$$. Should we run the approach above two times, i.e. after $$t_0$$ draws calculate $$S_{\theta}$$ and $$S_{\tau}$$ across the draws of $$\theta$$ and $$\theta$$ respectively? Are there any results on this?

• I'd reckon that just as Metropolis-within-Gibbs leads to multiple Metropolis-Hastings algorithms implemented in serial because you can't exploit the conditional dependence, you'd want to optimize the individual proposal distributions if you work under similar circumstances. – Forgottenscience Jul 29 '20 at 18:30
• Interesting. I'll have a look around. I've said this on here before, but check the new paper by Titsias and Dellaportas on adapative MCMC - I think its leagues better and much more promising compared to the ol' Haario method. – Forgottenscience Jul 29 '20 at 20:05
• Could I ask you to explain how you know the conditional distributions, but not the joint distribution? It would also be useful to say which dimensions $(\theta, \tau)$ are. – πr8 Jul 30 '20 at 10:40
• Note that when the full conditionals are available, the joint is as well, thanks to the Hammersley-Clifford theorem:$$\pi(\theta,\tau|D)\propto\dfrac{\pi(\theta|\tau_0,D)\pi(\tau|\theta,D)}{\pi(\theta_0|\tau_0,D)\pi(\tau_0|\theta,D)}$$ – Xi'an Jul 30 '20 at 12:45
• You need to adapt less and less often for the procedure to be valid. – Xi'an Jul 30 '20 at 14:46