When doing variational inference, due to intractability we typically maximize the evidence lower bound (ELBO) instead of minimizing Kullback-Leibler divergence (KLD) between our approximate and exact posterior. Assuming that we can compute gradients of KLD, my question is the following: are gradients of ELBO and KLD evaluated at the same values of our variational parameters pointing to the same direction (modulo sign) or can they be different.


1 Answer 1


Let $p(\theta \mid x)$ be the true posterior and $q_\phi(\theta)$ be the variational distribution (parameterized by $\phi$). The ELBO $\mathcal{L}(\phi)$ can be written as the difference between the log evidence and the KL divergence between the variational distribution and true posterior:

$$\mathcal{L}(\phi) = \log p(x) - D_{KL} \Big( q_\phi(\theta) \parallel p(\theta \mid x) \Big)$$

Take the gradient of both sides w.r.t. the variational parameters. The log evidence is constant, so $\nabla_\phi \log p(x) = 0$ and:

$$\nabla_\phi \mathcal{L}(\phi) = -\nabla_\phi D_{KL} \Big( q_\phi(\theta) \parallel p(\theta \mid x) \Big)$$

So, the gradients of the ELBO and KL divergence are opposites.

  • $\begingroup$ Does your notation $q_\phi(\theta)$ imply that $q$ is independent of the data $x$? (I'm also learning about this, hope you don't mind the question.) I thought that $q$ was updated based on $x$ $\endgroup$
    – 900edges
    Mar 15, 2021 at 15:39
  • 1
    $\begingroup$ @900edges The way I've written things here, $x$ is the data, $\theta$ are model parameters, and the approximate posterior $q_\phi(\theta)$ is completely determined by the variational parameters $\phi$. The variational parameters are learned from the data. In particular, we choose them to maximize the ELBO $\mathcal{L}(\phi)$, which is a lower bound on the log marginal likelihood $\log p(x) = \int p(x \mid \theta) p(\theta) d\theta$. So, $q_\phi(\theta)$ is indeed updated based on $x$. $\endgroup$
    – user20160
    Mar 15, 2021 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.