Consider the Gibbs sampler
- Sample $\theta' \sim p(\theta|\tau, D)$
- Sample $\tau' \sim p(\tau|\theta', D)$
where $\theta,\tau$ parameters of the data $D$. Now assume that we can only sample from $p(\tau|\theta', D)$ using a Metropolis step, for example because its normalization constant is unknown but we can evaluate the unnormalized density. In the Metropolis algorithm a proposal sample is drawn from a proposal distribution, such as the normal distribution. The step size is in this case determined by the variance of the normal, which is a tuning parameter for the acceptance rate of the Metropolis sampler.
At a given step in the Gibbs sampler, we could sample a large number of times from $p(\tau|\theta', D)$ i.e. holding $\theta'$ constant across repeated draws. Then we could estimate the acceptance rate by checking which proportion of proposals was accepted. However in the Metropolis-within-Gibbs sampler suggested above it is not common to draw a large number of samples from $p(\tau|\theta', D)$ at a given step; instead one sample is drawn by Metropolis and then the Gibbs sampler continues at the next iteration. Suppose that this next iteration yields $\theta''$ (two dashes), then the next Metropolis step in the second iteration will sample from $p(\tau|\theta'', D)$.
Now my specific question is: the optimal tuning parameters for the proposal distributions used for sampling from $p(\tau|\theta', D)$ at the first and $p(\tau|\theta'', D)$ at the second iteration may indeed be different. Assuming I can only set the tuning parameter once, how should I monitor acceptance rates in this Metropolis-Gibbs sampler? Is it enough to do this once across draws of $\tau$?