Bayesian MCMC: use the burn-in phase to find an appropriate scale factor for the likelihood?

Question

In a previous question I asked if I could scale the likelihood as my MCMC process advanced, to keep the acceptance fraction within a reasonable range (~0.2-0.5). I was told that this is not a valid approach, since doing that meant that the "Markov chain is no longer time homogeneous".

But, what if I used the burn-in stage to find an appropriate scale factor for my likelihood such that the acceptance fraction is reasonable? By "scale factor", I mean simply a real value that multiplies (and thus scales) my likelihood:

lkl_scaled = scale_factor * lkl_original

In this case I wouldn't be changing this factor during the MCMC process from which I later obtain the distributions of the model parameters. I would only do so during the burn-in phase, which I later discard.

I've tried this already and the results are excellent (where the chains get stuck forever with my original likelihood, they properly explore the parameters space with the scaled likelihood). I can't really see nothing wrong with this approach, but I'd like to be sure.

Is this a valid approach? If so, are there any caveats I should be aware of?

---

PD: I am aware of the existence of parallel tempering MCMC, but this approach is far simpler and it allows me to use other MCMC methods that would otherwise be of no use since the acceptance fraction is generally below 1%.

Answer 1

To give an idea for what the effect of doing this is, consider the following simple Gaussian model. $$ Y_i = \mu + \epsilon_i, \qquad \epsilon_i \sim N(0,1). $$ Suppose we put a flat prior on $\mu$. The genuine posterior in this case is $[\mu \mid Y_1, \ldots, Y_n] \sim N(\bar Y, n^{-1})$. Let's compare this to the posterior obtained from scaling the log-likelihood. $$ \pi_c(\mu \mid \mathbf Y) \propto \left[\prod_{i = 1}^N e^{-(Y_i - \mu)^2 / 2}\right]^c = \prod_{i = 1}^N e^{-c(Y_i - \mu)^2 / 2}. $$ By the usual argument, this leads to a Gaussian posterior for $\mu$ as well: $$ \mu \sim N(\bar Y, (cn)^{-1}). $$

Now, what conclusions can we draw?

• We are certainly not drawing from the genuine posterior.

• In this particular case, the posterior mean turns out to be the same. This suggests that, for point estimation, there are at least certain situations in which scaling the log-likelihood does not lead to a disaster.

• The scale factor is weakening our precision, and essentially corresponds to the posterior "throwing data away." For example, setting $c = 0.5$ is sort of like using half of the data.

• This "throwing away data" idea does not affect the posterior mean in this case because we used a flat prior, but would have also impacted the posterior mean if we had used (say) $\mu \sim N(0,\tau)$; in particular, we would have gotten more shrinkage towards zero.

• Even though we - for this particular example - have still got the correct posterior mean, the posterior variance is off. So, if I take $c$ to be very small to improve mixing, I should expect to be artificially inflating the variance by a factor of $1/c$.

The main concern with using this approach, and tuning $c$ to get a good acceptance rate, is that you will need to take $c$ very small. This is guaranteed to mess up your posterior variance, and is likely (but, evidently, not guaranteed) to mess up the posterior mean as well.

A more subtle problem if you do have weakly-informative priors is that, as the sample size grows, you will probably need to take $c$ going to $0$ to maintain a reasonable acceptance rate. This will cause the bias induced by the use of a weakly-informative prior to never be washed away by the data, so viewing this approach by embedding it in a standard Frequentist asymptotic analysis suggests that you may end up with inconsistent estimates.