Suppose that we have a very large i.i.d. sample $x_1,...,x_n$ and a data likelihood defined by $$p(x | \theta,\beta) = \prod_ip(x_i | \theta,\beta)$$. Further suppose that $\theta$ is the parameter of interest and that we have a very informative prior on it $p(\theta)$. $\beta$ are a set of nuisance parameters for which we have no great intuition about what the prior $p(\beta)$ should be, though it is reasonable to consider the prior to be diffuse.

One could of course perform Gibbs sampling on the full model to get a sample from the posterior, however I've been thinking about alternatives since computational complexity is a concern.

Consider instead observing $$ z(\theta) = \hat{I}^{1/2}(\hat{\theta} - \theta) \sim N(0,1) $$ where $\hat{\theta}$ is the MLE and $\hat{I}$ is the estimated Fisher information. We then have $$ p(\theta | z) \propto p(z(\theta) | \theta) p(\theta). $$ If $p(\theta)$ is distributed normally, then the posterior has a closed form normal solution. Further, the Bayesian central limit theorem assures us that the posterior using $z$ approaches the full data posterior.

I've been looking in the literature for a reference doing something like this but have come up empty. Can anyone point me to either a reference or a flaw in my thinking?

  • 2
    $\begingroup$ This is called the Laplace approximation and there is quite a decent literature on the topic. $\endgroup$ – Xi'an May 23 '16 at 18:00
  • $\begingroup$ Is it? The approximation here is to the data likelihood rather than the posterior, which is what the Laplace approximation typically operates on in my understanding. For example, is $p(\theta)$ is non-normal, the posterior using $z$ will also be non-normal even though we approximated the data likelihood as normal. $\endgroup$ – rasta May 23 '16 at 18:55
  • $\begingroup$ @Xi'an you might have missed the comment above. $\endgroup$ – Tim Mar 15 '17 at 15:57
  • $\begingroup$ @Tim: thanks, I also missed your comments! Correct point from rasta. However, the Normal approximations to the likelihood [classical theory] and to the posterior [Bernstein-von mises] are essentially equivalent in my opinion. $\endgroup$ – Xi'an Aug 12 '18 at 19:03
  • 1
    $\begingroup$ Thanks @Xi'an. Your second point is well taken. The posterior is as if only the estimates were observed, and not the full data. I'd challenge the first point though, as approximations of the data and posterior likelihoods can lead to quite different behavior. Consider for example a spike and slab prior where the posterior may be very non-normal even when $z$ is well approximated as normal. I did eventually find a (rather esoteric) reference: Kwan, Y. K. (1999). Asymptotic Bayesian analysis based on a limited information estimator. Journal of Econometrics, 88(1), 99-121. $\endgroup$ – rasta Aug 13 '18 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.