MCMC: invalid covariance matrix due to numerical error

During Gibbs sampling, I need to sample a multivariate Gaussian distribution. But rmvnorm return errors sometimes because the covariance matrix is not definite and I found the determinant of it is, say, $-10^{-20}$. I suspect this is caused by the numerical errors. What should I do to handle this?

• Why the covariance matrix is not definite? How did the errors occur? Could you tell us more? – Tim Apr 2 '17 at 8:53
• @Tim it can happen eg because there are some terms of the Gaussian with negligible coefficients. You could reduce the dimensionality of the covariance matrix to compensate, but in general that is much more complicated than the very much simpler and effective solution I have detailed in my answer below. – Hugh Perkins Apr 2 '17 at 12:47

You can add a tiny epsilon to the diagonal, say $1e-8$ or similar, to the covariance matrix.

So, let's say your covariance matrix is $\mathbf{\Sigma}$. And when you do stuff that involves $\text{inv}(\mathbf{\Sigma})$, or similar, it doesn't work very well, because your $\Sigma$ is not positive definite. So, therefore you can update your $\mathbf{\Sigma}$ as follows:

$$\mathbf{\Sigma}' = \mathbf{\Sigma} + 10^{-8} \mathbf{I}$$

Now your new $\mathbf{\Sigma}'$ matrix is positive definite, and your numerical calculations will work much more smoothly :-)

I've used this technique in the past, and it works very well for me.

• This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? You can also turn it into a comment. – gung Apr 2 '17 at 12:18
• Fleshed out the answer to explain how to update $\Sigma$ with the 1e-8, and the statement that this makes the matrix become positive definite. – Hugh Perkins Apr 2 '17 at 12:44
• @gung Thank you for your clear suggestion for how to improve my reply. Much appreciated :-) – Hugh Perkins Apr 2 '17 at 12:54