Does anyone know of any references that look at the relationship between the Laplace approximation and variational inference (with normal approximating distributions)? Namely I'm looking for something like conditions on the distribution being approximated for the two approximations to coincide.

  • $\begingroup$ Is your question about the computational accuracy of using the Laplace approximation vs say MCMC to estimate the posterior for latent variables in a multivariate normal setting? I don't understand the title: are we comparing them to each other or to something else? $\endgroup$ – AdamO Dec 26 '17 at 20:52
  • $\begingroup$ No I'm asking when the approximating distributions you get from the Laplace approximation and variational inference (using normal approximating distributions), which will both be normal, coincide. $\endgroup$ – aleshing Dec 26 '17 at 20:57
  • 1
    $\begingroup$ It is a limitation of my terminology, but I think what you call variational inference is also called Bayesian inference. Havard Rue at Norway has done work on nested Laplace transforms to approximate Variational Bayesian Inference. This has received considerable traction from the Bayesian community. They seem to coincide closely from what I've gathered of the work. onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2008.00700.x/… $\endgroup$ – AdamO Dec 26 '17 at 22:00
  • 2
    $\begingroup$ Variational inference (en.wikipedia.org/wiki/Variational_Bayesian_methods) refers to methods of approximate bayesian inference (like MCMC) where one approximates the posterior distribution by minimizing the KL divergence (or some other divergence) between the posterior and some distribution in an approximating family (unlike MCMC). INLA also preforms approximate inference, but takes an approach using the Laplace approximation (and it's restricted to a specific set of models, latent Gaussian models). This isn't that related to my question. $\endgroup$ – aleshing Dec 26 '17 at 22:15
  • $\begingroup$ I read this this paper a while back. It doesn't specifically address the Laplace approximation, but does give some results on asymptotic normality of VB approximations. It's kind of old. Generally the approaches for VB are pretty problem specific, and often the math is kind of complicated/tedious. For instance, this Annals paper gives some interesting results, but it's a really specific problem, and the hoops they had to jump through were immense. $\endgroup$ – Ryan Warnick Dec 31 '17 at 6:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.