I'm reading two descriptions of the VQ-VAE objective:

Kingma claims in page 18 that we want to maximize the ELBO, and shows that it can be written as $ELBO = logp_{\theta}(x) - KL(q_{\phi}(z|x)||p_{\theta}(z|x))$, the marginal likelihood of the data - the KL divergence between the approximate posterior and the true posterior of the latent variables.

Rocca claims that we just want to minimize the same $KL(q_{\phi}(z|x)||p_{\theta}(z|x))$ Kingma mentioned, but doesn't mention ELBO or the marginal likelihood of the data.

Are they saying the same thing? if so, why? is it because the marginal likelihood of the data is not a function of our variational parameters $\phi$? which is the "true" objective and which is just a math development? I need a good intuition on what is it that we're optimizing here...

  • 1
    $\begingroup$ these are equivalent, and you're exactly right: it's because adding a constant (wrt the variational parameters) doesn't change the optimization. $\endgroup$ Oct 19, 2022 at 18:38
  • $\begingroup$ @JohnMadden perhaps you could expand & elaborate on this comment in an answer. In particular, it would help future readers to understand why the $\log p_\theta(x)$ term is constant. $\endgroup$
    – Sycorax
    Oct 19, 2022 at 18:49

1 Answer 1


Let's consider $$ \begin{align} \left({\phi}^{*},{\theta}^{*}\right) &\in \operatorname*{arg\,max}_{\phi,\,\theta} \, \operatorname{ELBO}\left(\phi,\theta\right) \\&= \operatorname*{arg\,min}_{\phi,\,\theta} \, \left\{-\operatorname{ELBO}\left(\phi,\theta\right)\right\} \\&= \operatorname*{arg\,min}_{\phi,\,\theta} \, \left\{\operatorname{KL}\left(q_\phi\left(z\right)\|\ p_\theta\left(z|x\right)\right) - \ln\left(p_\theta\left(x\right)\right)\right\} \end{align} $$ for optimal parameters ${\theta}^{*}$ and ${\phi}^{*}$ with an asscociated optimal variational density $q_{\phi^*}\!\left(z\right)$.

In many Bayesian settings, where $\theta \equiv z$ is considered as random vector, an approximation to the posterior density $p_\theta\left(z|x\right) \equiv p\left(\theta|x\right)$ is sought and $p_\theta\left(x,z\right) \equiv p\left(x,\theta\right)$ is the product of the prior density $p_\theta\left(z\right) \equiv p\left(\theta\right)$ and the density of the observation model/likelihood $p_\theta\left(x|z\right) \equiv p\left(x|\theta\right)$. In this case, the ELBO and KL-divergence become independent of $\theta$ and maximizing $\operatorname{ELBO}\left(\phi\right)$ w.r.t. $\phi$ is equivalent to minimizing the KL-divergence w.r.t. $\phi$ since $p_\theta\left(x\right)$ is by assumption constant w.r.t. $\phi$. The KL-divergence is a common measure of discrepancy between two probabilit densities (but not a metric) which is non-negative and attains zero iff they coincide almost everywhere. One idea could thus be to find an "optimal" approximation $q_{\phi^*}\!\left(\theta\right)$ to the posterior density by minimizing the KL-divergence between $q_\phi\left(\theta\right)$ and $p\left(\theta|x\right)$ w.r.t. $\phi$. However, minimizing the KL-divergence directly is assumed to be inadmissible as it depends on the true posterior density, which is unknown or intractable (otherwise there would be no need to find an approximation to it in the first place). But in the evidence lower bound $\operatorname{ELBO}\left(\phi\right) \equiv \int \ln\left(p\left(x,\theta\right)/q_\phi\left(\theta\right)\right)q_\phi\left(\theta\right)\mathrm d \theta$, we are no longer dealing with $p\left(\theta|x\right)$ but with the joint density $p\left(x,\theta\right)$ which is assumed to have a known functional form.

In different settings, when dealing with models where the likelihood $p_\theta\left(x|z\right)$ is conditioned on a latent vector $z$, the "ELBO" is a log-likelihood lower bound. Now, by maximizing this lower bound, given by $\operatorname{ELBO}\left(\phi,\theta\right) \equiv \int \ln\left(p_\theta\left(x,z\right)/q_\phi\left(z\right)\right)q_\phi\left(z\right)\mathrm d z$, over $\phi$ and $\theta$, the goal is to both minimize the KL-divergence between $q_\phi\left(z\right)$ and $p_\theta\left(z|x\right)$ and maximize the approximate log-likelihood. Here, ${\theta}^{*}$ is a variational approximation to the MLE.

Ormerod, J. T., & Wand, M. P. (2010). Explaining variational approximations. The American Statistician, 64(2), 140-153.

  • $\begingroup$ thanks! so if the ELBO itself is tractable - why does rocca show that we are optimizing the KL divergence? he shows that we can develop the KL divergence between the approximate posterior and the true posterior (which is indeed unknown) as a sum of the data likelihood and the KL divergence between the approximate posterior and the prior, and then proceeds to optimize that. From your answer I understand that you're implying that we can directly optimize the expression you gave for the ELBO, why isn't it so in practice??? $\endgroup$
    – ihadanny
    Oct 20, 2022 at 5:48
  • $\begingroup$ @ihadanny I tried to go into a bit more detail now - hope this helps. It's quite common to motivate the optimization problem from the perspective of minimizing KL-divergence. In practice, algorithms like stochastic gradient ascent are used to solve the high-dimensional maximization problem (i.e., to find the high-dimensional maximizer of the ELBO). I'm not an expert on variational autoencoders, though. $\endgroup$
    – statmerkur
    Oct 20, 2022 at 12:36
  • 1
    $\begingroup$ @ihadanny Do you mean the expression $\mathbb{E}_{z\sim q_x}\left(\log p\left(x|z\right)\right)-KL\left(q_x\left(z\right),p\left(z\right)\right)$? If so, this is equal to $\int \log\left\{p\left(x|z\right)p\left(z\right)/q_x\left(z\right)\right\}q_x\left(z\right)\mathrm dz$, which (after using $p\left(x|z\right)p\left(z\right)=p\left(x,z\right)$ and translating between the notations) is exactly the same as the ELBO expressions I use. $\endgroup$
    – statmerkur
    Oct 20, 2022 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.