Techniques for improving mixing when sampling from a multidimensional posterior

Question

I'm currently trying to use the Metropolis-Hastings algorithm to sample from a posterior distribution of the form

$$p(\theta | y ) \propto \prod_{ij} \phi (\theta_{ij}) \times \prod_{i=1}^n \pi_{y_i}(x_i|\theta)$$

Where $\phi()$ is the standard normal density function. This posterior is the result of taking the multinomial logit model with multivariate normal prior $N_p (0, I_p)$.

I have taken the proposal distribution to be

$$q(\theta_{new} | \theta_{old}) \sim N_p (\theta_{old}, I_p)$$

As far as I can tell my implementation of the algorithm is correct, but mixing is incredibly poor. The acceptance ratio is consistently on the order of $1\times 10^{-3}$ or worse.

What are some techniques for improving mixing for sampling from a high dimensional posterior? If the dimension was low, I would plot the distribution to get an idea of what proposal distribution would suit it. But when the posterior is high dimensional, this doesn't seem feasible.

Answer 1

There is a vast literature associated to MCMC. So the good news is that there probably exists a technique which would be of help to you.

The bad news is that we would need to know more about the model you are trying to fit. For example, it might make sense to block the $\theta$ in various ways, do Gibbs sampling, various HMC or other gradient based techniques etc etc.

From the description already there, have you tried initialising the chain using the parameters from a regular multinomial logit fit?

Also note that random walk metropolis also does often have very low acceptance ratios.