I am training a complex Bayesian model using Gibbs sampling and Metropolis-Hasting algorithm. Most of the parameters are directly sampled by using conjugate priors except for 3 params which are sampled by M-H. In tuning the "step size" of random walk M-H, I found making the acceptance rate to be around 0.234~0.44 will only give 1%~5% effective samples (strong autocorrelations). May I ask is the "effective size" more "important" than the acceptance rate as a criterion? And which criterion should I "ultimately" care here?

  • $\begingroup$ What are the effective samples you get if acceptance rate it increased? $\endgroup$ Sep 28 '16 at 12:53
  • 1
    $\begingroup$ The effective samples has a lot more to do with the posterior correlation amongst your parameters than the acceptance rate of your MH steps, e.g. using a Gibbs sampler on a bivariate normal with high correlation will result in a Markov chain that has high correlation in your samples even though the acceptance rate is 1. $\endgroup$
    – jaradniemi
    Sep 28 '16 at 14:41
  • 1
    $\begingroup$ Aiming at this "optimal" acceptance rate in a regular random walk Metropolis-Hastings algorithm is equivalent to minimising the asymptotic variance of the resulting estimator, hence at reducing the autocorrelation in the chain. The effective sample size is a poor proxy with well-known shortcomings and is too easily adopted due to its simplicity. $\endgroup$
    – Xi'an
    Sep 29 '16 at 20:02
  • $\begingroup$ I am quite new to this. Which paper/book are the best in discussing this topics? I am currently reading the source code of jags. It seems it just use acceptance rate as the criterion. And the adaptive alg is "noisy gradient" --- what is that about? I cannot find any references on this. $\endgroup$
    – user112758
    Sep 29 '16 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.