Bayesian MLPs using the MCMC methods - any tricks of the trade? Having used the NETLAB library for MATLAB to implement Bayesian Multi-Layer Perceptron (MLP) neural networks using MacKay's evidence framework, I am now experimenting with Markov Chain Monte Carlo (MCMC) methods using metrop() and hmc().  Does anyone have any advice for successful modelling using this approach (preferably using NETLAB), e.g. how to set the parameters to get good mixing and acceptably low rejection ratios etc?
Has anybody used the Metrpolois-Hastings or HMC samplers for applications of MLP neural networks?  I'm beginning to think that perhaps everybody just uses the evidence framework!
 A: I have no experience relating to Bayesian Neural Nets and HMC, but with a related task.
That involved using neural nets as an unormalized probability model (i.e. an energy based model) where training involved generating "contrastive/negative samples" from the model distribution via HMC. I suppose the problem is somewhat similar, since both models share some factors. 
(More specifically, ${\partial y \over \partial w}$ where $w$ is the first layer's weights and $y$ the output of the net. Bayesian neural nets have a $\partial L / \partial y$ in front, EBMs a different $\partial E / \partial y$ in front and a $\partial w / \partial  x = w$ at the end.)
Long story short: to get decent results using adaptive step rates to reach a desired acceptance rate was making a huge difference.
You can find some code here, but it is python. The approach is described in "Learning Deep Energy Models", Ngiam et al and there is a footnote on the exact settings for HMC.
A: Radford Neal's flexible Bayesian modeling toolkit (fbm) also allows you to play with the Monte Carlo methods and has neural network functionality built in.
