I've read a bit about how to derive parametrization-invariant priors for models where we have access to derivatives of the likelihood function and can compute the Fisher Information Matrix: http://www.philipgoyal.org/resources/Papers/Research-Papers/MaxEnt2005.pdf

Is there a good way to derive roughly objective priors for simulator-based models, when we don't have access to derivatives of the likelihood function?

So far I've just been using heavy-tailed priors and hoping for the best, but that doesn't seem like the best approach.


A description length based prior will have the properties you seek.

To see the connection, imagine that you've fitted a continuous model $p(x|\theta)$ with maximum likelihood. You'd now like to encode $x$ using $\theta$. A simple way to encode the vector $\theta$ is as an array of floating point numbers with fixed precision. Suppose in doing so you introduce an error of size $\epsilon \sim 2^{-p}$ for each coordinate. Since $\theta$ is a MLE estimate, this will create a second order effect on $p(x|\theta)$ that will depend on the hessian of the likelihood function and $\epsilon^2$. If the function is peaked enough (or a member of the exponential family) this is approximately going to be... the Fisher Information Matrix! Dum dum dum...

  • $\begingroup$ Arthur, could you expand a bit on this? Do you have a reference for this? Would this provide the Fisher Information around the MLE or also across the parameters space? (we want the latter, right?). How would we implement this practically in am MCMC? $\endgroup$ – Florian Hartig Aug 18 '15 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.