Classical variational inference uses mean field theory because of its computational benefits, i.e. assumes that the latent variables are independent. For gaussian distribution, it wants to find a variational normal distribution with diagonal covariance matrix which is close enough to the standard normal distribution as the prior density.

What is the the form of evidence lower bound (ELBO) for an especial case with dependent latent variables? As we know, in mean field family, ELBO contains two parts, a KL divergence (generally between a normal distribution with diagonal covariance matrix and the standard normal distribution) and the reconstruction error. If we assume that our latent variables have a multivariate normal distribution with a full covariance matrix(that shows they are dependent), how does the model change? (I know what is the formula for KL divergence between normal distributions, but it seems the variational ELBO should involve some additional terms). Thanks beforehand.

  • $\begingroup$ When you say "How does the model change now when we have a full covariance matrix?", what are you referring to? $\endgroup$ – πr8 Sep 19 '19 at 22:38
  • $\begingroup$ Since I mentioned to normal distribution in the question, I mean assume our latent variables have a multivariate normal distribution with full covariance matrix, and not only a diagonal covariance matrix. It shows the dependency between the latent variables. $\endgroup$ – Ham82 Sep 19 '19 at 22:42
  • $\begingroup$ Right - I understand that you are choosing to use a full covariance matrix in your variational family. It is not clear to me which model you're referring to when you ask "how does the model change?". $\endgroup$ – πr8 Sep 19 '19 at 22:44
  • $\begingroup$ You are right, the question was not clear. I modified it. I want to know how can I find the new formula for ELBO? I think it is not only put the new normal distribution to the KL divergence part to find the difference between that and the standard normal distribution. $\endgroup$ – Ham82 Sep 19 '19 at 22:50

The ELBO is, as usual, given by

\begin{align} \mathcal{L} ( q ) &= \log p ( x ) - KL ( q ( z ) || p ( z | x) ) \\ &= \int q ( z ) \log \frac{ p ( x, z ) }{ q ( z )} dz \\ &= \mathbf{E}_q \left[ \log p ( x, z ) \right] + \mathcal{H} ( q ) \end{align}

where $\mathcal{H}$ is the entropy.

Now, if $q ( z ) = \mathcal{N} ( \mu, \Sigma )$, one has that $ \mathcal{H} ( q ) = \frac{1}{2} \log \det \left( 2 \pi e \Sigma \right)$.

Moreover, if $\xi \sim \mathcal{N} ( 0, I )$, then $\mu + \Sigma^{1/2} \xi$ is distributed as $q$. As a result, one can write

\begin{align} \mathbf{E}_q \left[ \log p ( x, z ) \right] = \mathbf{E}_{\xi \sim \mathcal{N} ( 0, I )} \left[ \log p ( x, \mu + \Sigma^{1/2} \xi ) \right]. \end{align}

This allows us to write the ELBO as a function of $( \mu, \Sigma )$:

\begin{align} \mathcal{L} ( \mu, \Sigma ) &= \mathbf{E}_{\xi \sim \mathcal{N} ( 0, I )} \left[ \log p ( x, \mu + \Sigma^{1/2} \xi ) \right] + \frac{1}{2} \log \det \left( 2 \pi e \Sigma \right). \end{align}

One can estimate gradients of the first term by noting that

\begin{align} \nabla_{ ( \mu, \Sigma) } \mathbf{E}_{\xi \sim \mathcal{N} ( 0, I )} \left[ \log p ( x, \mu + \Sigma^{1/2} \xi ) \right] = \mathbf{E}_{\xi \sim \mathcal{N} ( 0, I )} \left[ \nabla_{ ( \mu, \Sigma) } \log p ( x, \mu + \Sigma^{1/2} \xi ) \right] \end{align}

and thus gradients can be estimated unbiasedly by Monte Carlo. Gradients of the second term can be obtained analytically.

In closing, one can compute unbiased estimates of the gradient of $\mathcal{L} ( \mu, \Sigma )$, which allows for efficient gradient-based optimisation of the ELBO. This is roughly the idea behind Automatic Differentiation Variational Inference.

| cite | improve this answer | |
  • $\begingroup$ Thanks for this nice answer and also the reference paper. It is exactly what I looked for. $\endgroup$ – Ham82 Sep 19 '19 at 23:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.