# Comparing Laplace Approximation and Variational Inference

Does anyone know of any references that look at the relationship between the Laplace approximation and variational inference (with normal approximating distributions)? Namely I'm looking for something like conditions on the distribution being approximated for the two approximations to coincide.

Edit: To give some clarification, suppose you want to approximate some distribution with density $$f(\theta)$$ which you only know up to proportionality. When using the Laplace approximation, you approximate it with the density of a Normal distribution with mean $$\hat{\mu}_1$$ and covariance $$\hat{\Sigma}_1$$ where $$\hat{\mu}_1=\arg \max_{\theta}f(\theta)$$ and $$\hat{\Sigma}_1=[-\nabla\nabla \log f(\theta)\mid_{\theta=\hat{\mu}}]^{-1}$$. When using variational inference with a normal approximating distribution, you approximate it with the density of a Normal distribution with mean $$\hat{\mu}_2$$ and covariance $$\hat{\Sigma}_2$$, where $$(\hat{\mu}_2,\hat{\Sigma}_2)=\arg \max_{(\mu,\Sigma)}KL(\phi_{(\mu,\Sigma)}||f)$$, $$KL$$ is the KL-Divergence, and $$\phi_{(\mu,\Sigma)}$$ denotes a Normal density with mean and covariance $$(\mu,\Sigma)$$. Under what conditions do we have $$(\hat{\mu}_1,\hat{\Sigma}_1)=(\hat{\mu}_2,\hat{\Sigma}_2)$$?

• Is your question about the computational accuracy of using the Laplace approximation vs say MCMC to estimate the posterior for latent variables in a multivariate normal setting? I don't understand the title: are we comparing them to each other or to something else? Commented Dec 26, 2017 at 20:52
• It is a limitation of my terminology, but I think what you call variational inference is also called Bayesian inference. Havard Rue at Norway has done work on nested Laplace transforms to approximate Variational Bayesian Inference. This has received considerable traction from the Bayesian community. They seem to coincide closely from what I've gathered of the work. onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2008.00700.x/… Commented Dec 26, 2017 at 22:00
• Variational inference (en.wikipedia.org/wiki/Variational_Bayesian_methods) refers to methods of approximate bayesian inference (like MCMC) where one approximates the posterior distribution by minimizing the KL divergence (or some other divergence) between the posterior and some distribution in an approximating family (unlike MCMC). INLA also preforms approximate inference, but takes an approach using the Laplace approximation (and it's restricted to a specific set of models, latent Gaussian models). This isn't that related to my question. Commented Dec 26, 2017 at 22:15
• I read this this paper a while back. It doesn't specifically address the Laplace approximation, but does give some results on asymptotic normality of VB approximations. It's kind of old. Generally the approaches for VB are pretty problem specific, and often the math is kind of complicated/tedious. For instance, this Annals paper gives some interesting results, but it's a really specific problem, and the hoops they had to jump through were immense. Commented Dec 31, 2017 at 6:35
• shouldn't it be argmin for the KL divergence? (instead of argmax) Commented Aug 28, 2020 at 15:04

I am not aware of any general results, but in this paper the authors have some thoughts for Gaussian variational approximations (GVAs) for generalized linear mixed model (GLMMs). Let $$\vec y$$ be the observed outcomes, $$X$$ be a fixed effect design matrix, $$Z$$ be a random effect design, denote an unknown random effect $$\vec U$$, and consider a GLMM with densities:

\begin{align*} f_{\vec Y\mid\vec U} (\vec y;\vec u) &= \exp\left(\vec y^\top(X\vec\beta + Z\vec u) - \vec 1^\top b(X\vec\beta + Z\vec u) + \vec 1^\top c(\vec y)\right) \\ f_{\vec U}(\vec u) &= \phi^{(K)}(\vec u;\vec 0, \Sigma) \\ f(\vec y,\vec u) &= f_{\vec Y\mid\vec U} (\vec y;\vec u)f_{\vec U}(\vec u) \end{align*}

where I use the same notation as in the paper and $$\phi^{(K)}$$ is a $$K$$-dimensional multivariate normal distribution density function.

### Using a Laplace Approximation

Let

$$g(\vec u) = \log f(\vec y,\vec u).$$

Then we use the approximation

$$\log\int \exp(g(\vec u)) d\vec u \approx \frac K2\log{2\pi - \frac 12\log\lvert-g''(\widehat u)\rvert} + g(\widehat u)$$

where

$$\widehat u = \text{argmax}_{\vec u} g(\vec u).$$

### Using a Gaussian Variational Approximation

The lower bound in the GVA with a mean $$\vec\mu$$ and covariance matrix $$\Lambda$$ is:

\begin{align*} \int \exp(g(\vec u)) d\vec u &\approx \vec y^\top(X\vec\beta + Z\vec\mu) - \vec 1^\top B(X\vec\beta + Z\vec\mu, \text{diag}(Z\Lambda Z^\top)) \\ &\hspace{25pt}+ \vec 1^\top c(\vec y) + \frac 12 \Big( \log\lvert\Sigma^{-1}\rvert + \log\lvert\Lambda\rvert -\vec\mu^\top\Sigma^{-1}\vec\mu \\ &\hspace{25pt} - \text{trace}(\Sigma^{-1}\Lambda) + K \Big) \\ B(\mu,\sigma^2) &= \int b(\sigma x + \mu)\phi(x) d x \end{align*}

where $$\text{diag}(\cdot)$$ returns a diagonal matrix.

### Comparing the Two

Suppose that we can show that $$\Lambda\rightarrow 0$$ (the estimated conditional covariance matrix of the random effects tends towards zero). Then the lower bound (disregarding a determinant) tends towards:

\begin{align*} \int \exp(g(\vec u)) d\vec u &\approx \vec y^\top(X\vec\beta + Z\vec\mu) - \vec 1^\top b(X\vec\beta + Z\vec\mu) \\ &\hspace{25pt}+ \vec 1^\top c(\vec y) + \frac 12 \Big( \log\lvert\Sigma^{-1}\rvert -\vec\mu^\top\Sigma^{-1}\vec\mu + K\Big) \\ &= g(\vec\mu) + \dots \end{align*}

where the dots do not depend on the model parameters, $$\vec\beta$$ and $$\Sigma$$. Thus, maximizing over $$\vec\mu$$ yields $$\vec\mu\rightarrow \widehat u$$. Then the only difference between the Laplace approximation and the GVA is a

$$- \frac 12\log\lvert -g''(\widehat u)\rvert$$

term. We have that

$$-g''(\widehat u) = \Sigma^{-1} + Z^\top b''(X\vec\beta + Z\vec u)Z$$

where the derivatives are with respect to $$\vec\eta = X\vec\beta + Z\vec u$$. This does not tend towards zero as the conditional distribution of the random effects becomes more peaked. However, still very hand wavy, it may cancel out with the

$$\frac 12\log\lvert\Lambda\rvert = -\frac 12\log\lvert\Lambda^{-1}\rvert$$

term we disregarded in the lower bound. The first order condition for $$\Lambda$$ is:

$$\Lambda^{-1} = \Sigma^{-1} + Z^\top B^{(2)}(X\vec\beta + Z\vec\mu, \text{diag}(Z\Lambda Z^\top)Z$$

where

$$B^{(2)}(\mu,\sigma^2) = \int b''(\sigma x+ \mu)\phi(x) dx.$$

Thus, if $$\vec\mu \approx \widehat u$$ and $$\Lambda \approx 0$$ then:

$$\Lambda^{-1} \approx \Sigma^{-1} + Z^\top b''(X\vec\beta + Z\vec u)Z$$

and the Laplace approximation and the GVA yield the same approximation of the log marginal likelihood.

### Notes

Do also see the annals paper Ryan Warnick mentions.

There's a nice old Neural Computation paper on the relationship between the Laplace approximation and variational inference with a Gaussian proxy posterior:

http://www0.cs.ucl.ac.uk/staff/c.archambeau/publ/neco_mo09_web.pdf

In fine, the variational approximation is equivalent to requiring the Laplace approximation to hold on average, where the average is taken under the proxy posterior, as opposed to just "locally." Thus, the mean of the proxy posterior under a Laplace approximation is the point (assuming there's only one) where the gradient of the true log-posterior is zero; whereas the mean of the proxy posterior under the variational Gaussian approximation is the point that renders the average of the gradient of the true log-posterior zero. Similarly for the covariance matrix.

• This sounds interesting, I am closer to understanding it. Could you please elaborate on what the "proxy posterior" is? Commented Nov 3, 2022 at 9:40
• @ngiann "proxy posterior" = the approximate posterior used for inference. The Laplace approximation to the posterior (i.e., the "proxy posterior") is $N(\hat{\mu}, \hat{\Sigma})$, with $\hat{\mu}$ and $\hat{\Sigma}$ calculated as in the OP. In variational inference, one chooses some parametric form for the (proxy) posterior. If one chooses it to be a Gaussian, then the result from Opper's paper holds. Commented Nov 4, 2022 at 13:34
• ...The approximate/fake/proxy posterior is denoted $q$ in Opper's paper (and most other papers on variational inference). The result I described in my original reply can be found in the paper by comparing Eqs. 6, 7 (variational inference under a Gaussian) with Eq. 5 (the Laplace approximation). Commented Nov 4, 2022 at 13:49
• I see. The term proxy confused me. I typically call it the approximating posterior. Thanks for your clarification, I completely understand it finally! Commented Nov 5, 2022 at 16:41