Consider the Gibbs sampler
- Sample $\theta' \sim p(\theta|\tau, D)$
- Sample $\tau' \sim p(\tau|\theta', D)$
Both conditional distributions are sampled with a Metropolis step. The joint distribution is unknown so that only a Gibbs sampler with two separate Metropolis steps can be used.
There are many forms of adaptive Metropolis algorithms which change the covariance matrix $\Sigma$ of the proposal (jump-) distribution assumed to be MV-normal and centered at the current values: $N(x|x',\Sigma)$. For example, in this question I wrote down the adaptive MH algroithm of Haario et al (1999). In an extension Haario et al. (2001) suggested to evaluate the empirical covariance matrix $S$ over the first $t_0$ MCMC draws and scaled by a factor $s=2.4^2/d$ as proposal variance $\Sigma$ (with d the dimension). A similar strategy is suggested by Gelman et al in 'Bayesian Data Analysis'(3rd ed., p. 296).
Now, when there is only one posterior to sample from, the approach is clear. However what should one do in the case of the two MH steps within Gibbs? We then have two proposals $\Sigma_{\theta}$ and $\Sigma_{\tau}$. Should we run the approach above two times, i.e. after $t_0$ draws calculate $S_{\theta}$ and $S_{\tau}$ across the draws of $\theta$ and $\theta$ respectively? Are there any results on this?
