I'm trying to do model selection using PyMC (v2.2), but having difficulty assessing the models using various Information Criteria and/or Bayes Factor. My model is similar to a typical regression, with several parameters (~10) with priors modelled by uniform distributions, and a single observation modelled by a normal distribution with a deterministic mean and uniform standard deviation (through a precision deterministic). The mean (dynamic) response of the system is actually generated by an ordinary differential equation and typically yields around 1500 data points. I can get reasonably accurate results using either Adaptive or Non-Adaptive MCMC, with around 50K samples following 50K burn-in samples.
Based on issues reported with BF for complex models, I started looking at the DIC values produced after performing the MCMC analysis, but for models consisting of various combinations of the true parameter set, they were quite large with very little difference between them. For example, -14623.9 and -14624.8. Are the DIC, and other similar criterion such as BPIC, normally so insensitive to different (sub-)models?
Using some code from the sandbox, I also computed the sample likelihood for my models in an attempt to exploit Bayes Factor. However, the log-likelihoods produced were very large, ranging from -1e5 to 1e5. Regularization further decreased the minimum and exponentiating resulted in overflow (see function
weight), so the array
exp(loglikes[m]) ended up comprising all zeros and one one! Why would my log-likelihoods, calculated by
logp after a call to
draw_from_prior, be so large?
I'm fairly new to Bayesian estimation, so any help would be greatly appreciated!