Is there a standard technique to debug MCMC programs?

Question

Debugging MCMC programs is notoriously difficult. The difficulty arises because of several issues some of which are:

(a) Cyclic nature of the algorithm

We iteratively draw parameters conditional on all other parameters. Thus, if a implementation is not working properly it is difficult to isolate the bug as the issue can be anywhere in the iterative sampler.

(b) The correct answer is not necessarily known.

We have no way to tell if we have achieved convergence. To some extent this can be mitigated by testing the code on simulated data.

In light of the above issues, I was wondering if there is a standard technique that can be used to debug MCMC programs.

Edit

I wanted to share the approach I use to debug my own programs. I, of course, do all of the things that PeterR mentioned. Apart from those, I perform the following tests using simulated data:

1. Start all parameters from true values and see if the sampler diverges too far from the true values.

2. I have flags for each parameter in my iterative sampler that determines whether I am drawing that parameter in the iterative sampler. For example, if a flag 'gen_param1' is set to true then I draw 'param1' from its full conditional in the iterative sampler. If this is set to false then 'param1' is set to its true value.

Once I finish writing up the sampler, I test the program using the following recipe:

• Set the generate flag for one parameter to true and everything else to false and assess convergence with respect to true value.
• Set the generate flag for another parameter in conjunction with the first one and again assess convergence.

The above steps have been incredibly helpful to me.

Answer 1

Standard programming practice:

• when debugging run the simulation with fixed sources of randomness (i.e. same seed) so that any changes are due to code changes and not different random numbers.
• try your code on a model (or several models) where the answer IS known.
• adopt good programming habits so that you introduce fewer bugs.
• think very hard & long about the answers you do get, whether they make sense, etc.

I wish you good luck, and plenty of coffee!

Answer 2

I have a depressing and not-very-specific anecdote to share here. I spent some time as a co-worker of a statistical MT researcher. If you want to see a really big, complex, model, look no further.

He was putting me through NLP bootcamp for his own amusement. I am, in general, the sort of programmer who lives and dies by the unit test and the debugger. As a young person at Symbolics, I was struck by the aphorism, 'programming is debugging an empty editor buffer.' (Sort of like training a perceptron model.)

So, I asked him, 'how do you test and debug this stuff.' He answer was, "You get it right the first time. You think it through (in his case, often on paper) very carefully, and you code it very carefully. Because when you get it wrong, the chances of isolating the problem are very slim."

Answer 3

Good tips in PeterR's answer; I don't have any further tips for the actual debugging, but I found a very useful proceedure for testing if your code could have a bug. It's described in this paper:

http://pubs.amstat.org/doi/abs/10.1198/016214504000001132

Essentially the idea is to have two simulations: One being your MCMC for inferring (presumably) the parameters of your model. The second simulator simply samples parameters from the prior. They generate data from the parameters of both simulators, and calculate a test statistic comparing the joint distributions of parameters and data. If the MCMC code correctly samples parameters from the posterior, then the test statistic will have a distribution of N(0,1). Code for calculating the test statistic is available.