I currently am trying to figure out what would be the best option to perform MCMC sampling for a model which may show some kind of pathological behavior for some parameter combinations. The concrete case is a recurrent neural network (RNN), that I am running until convergence and then try to fit some values at the convergence point using MCMC. However, for some parameter combinations, the model may not converge or may produce some activations which are invalid.
That is, the MCMC should only sample from regions, where the parameters are valid and completely ignore invalid parameters.
The best option would, of course, be to use the priors to only allow sampling from those regions. However, this would require me to know the actual geometry of the valid region. While I currently have some intuition about this geometry (e.g. I expect it to be connected), I cannot give an analytical description, so I cannot constrain sampling in the prior.
Nevertheless, I can easily detect invalid parameters, after I run the model. For example, I detect divergent behavior by setting a maximum number of iterations and setting a flag when this number is exceeded.
Is it somehow possible to use this information to constrain the parameter space that is explored during sampling? The idea I currently have in mind is to use the flags that indicate pathological behavior as observed variables in a Bernoulli distribution with very low probability. This should reduce the likelihood of the sampler by drawing samples from this region of the parameter space and cause it to jump back to the allowed region.
Would this approach work in general, or could this cause additional artifacts? Also has this approach or a similar one been tried before, such that I can cite any literature on this approach?
I am currently using the Metroplis step method from pymc3. I also tried NUTS, but this turned out to be slower by a factor of about 10 compared to Metropolis. The reason is not yet clear, and I believe NUTS may speed up when I manage to constrain the problem correctly.