# Covariance matrix proposal distribution

In a MCMC implementation of hierarchical models, with normal random effects and a Wishart prior for their covariance matrix, Gibbs sampling is typically used.

However, if we change the distribution of the random effects (e.g., to Student's-t or another one), the conjugacy is lost. In this case, what would be a suitable (i.e., easily tunable) proposal distribution for the covariance matrix of the random effects in a Metropolis-Hastings algorithm, and what should be the target acceptance rate, again 0.234?

Thanks in advance for any pointers.

Well, if you are looking "for any pointers"...

The (scaled)(inverse)Wishart distribution is often used because it is conjugate to the multivariate likelihood function and thus simplifies Gibbs sampling.

In Stan, which uses Hamiltonian Monte Carlo sampling, there is no restriction for multivariate priors. The recommended approach is the separation strategy suggested by Barnard, McCulloch and Meng: $$\Sigma=\text{diag_matrix}(\sigma)\;\Omega\;\text{diag_matrix}(\sigma)$$ where $\sigma$ is a vector of std devs and $\Omega$ is a correlation matrix.

The components of $\sigma$ can be given any reasonable prior. As to $\Omega$, the recommended prior is $$\Omega\sim\text{LKJcorr}(\nu)$$ where "LKJ" means Lewandowski, Kurowicka and Joe. As $\nu$ increases, the prior increasingly concentrates around the unit correlation matrix, at $\nu=1$ the LKJ correlation distribution reduces to the identity distribution over correlation matrices. The LKJ prior may thus be used to control the expected amount of correlation among the parameters.

However, I've not (yet) tried non-normal distributions of random effects, so I hope I've not missed the point ;-)

• This answer talks about the prior, the OP asks about the proposal... Does these priors help with the acceptance ratio in some way? Jul 3, 2019 at 11:02
• @Sycorax What about the proposal that the OP asked? what should he use, and with what parameters? Jul 3, 2019 at 11:02

I personally use Wishart proposals. For instance, if I want a proposal $$\Sigma^*$$ around $$\Sigma$$, I use: $$\Sigma^* \sim \mathcal{W}(\Sigma/a,a),$$ where $$a$$ is a large number, like 1000. With that trick you will get $$E[\Sigma^*]=\Sigma$$ and you can adjust the variance with $$a$$. If I am not mistaken, the ratio of proposals for $$(p\times p)$$ matrices has a closed form: $$\frac{q(\Sigma\to\Sigma^*)}{q(\Sigma^*\to\Sigma)} = \left(\frac{|\Sigma^*|}{|\Sigma|}\right)^{a-(p-1)/2} \cdot e^{[tr({\Sigma^*}^{-1}\Sigma)-tr({\Sigma}^{-1}\Sigma^*)] \cdot a/2}$$

It is well known that if you use non-Gaussian distributions, the conjugacy of the model is lost, see:

http://www.utstat.toronto.edu/wordpress/WSFiles/technicalreports/0610.pdf

Then, you need to use other MCMC methods, such as Metropolis within Gibbs sampling or some adaptive version of it. Fortunately, there is an R package for doing so:

http://cran.r-project.org/web/packages/spBayes/index.html

The recommended acceptance rate is 0.44 but, of course, there are some assumptions behind this number, similarly as in the case of the 0.234.

Are you THE Dimitris Rizopoulos?

• @DimitrisRizopoulos The adaptive Metropolis withing Gibbs I mentioned uses a finite mixture of Gaussian distributions as a proposal distribution (as stated in the technical report I posted). If you use the hardcore Metropolis, then you are asking for an answer to the "million dollar question", for which there is no general solution. Typically you have to play with different proposals and different acceptance rates. Very good book, by the way.
– Teco
Nov 26, 2013 at 21:07

Any proposal can be used if you define your log-posterior properly. You just need to use some tricks to implement it and properly define the support of your posterior, see:

How to find the support of the posterior distribution to apply Metropolis-Hastings MCMC algorithm?

There are tons of examples where a Gaussian proposal can be used for truncated posteriors. This is just an implementation trick. Again, you are asking a question with no general solution. Some proposals even have different performance for the same model and different data sets.

Good luck.

• Well, taking into account that the covariance matrix needs to be positive definite, it does not seem that logical to me to use whatever proposal distribution. The proposed matrices need to be positive definite. One option would be to have as proposal the Wishart posterior conditional used in Gibbs sampling, however this did not seem to work particularly well when I assumed a student's-t for the random effects. Hence my question, are there other types of proposals for covariance matrices? Nov 27, 2013 at 19:45