2
$\begingroup$

I'm studying variational inference (in the context of VAEs), and I'm watching this video at this time point. At this point in the video, the goal of approximating the intractable posterior $p_{\theta}(z|x)$ by an approximate posterior $q_{\phi}(z|x)$ has been set, and the ELBO formulation has been presented.

enter image description here

As is shown on the image, what the author of the video says at this point is that, since marginal log-likelihood will not change, maximizing ELBO minimizes the KL divergence between the $q$ and $p$ distributions (which is what we want). To quote more precisely, the author says : "We cannot compute [marginal log-likelihood] but we know that it will not change and so we have found a way around our KL divergence".

This sentence confused me. In the context of variational auto-encoders for generating data, I thought that the goal was specifically to increase the evidence by finding the optimal $\theta$ (and $\phi$) parameters, and so that evidence $p_{\theta}(x)$ will be maximized during the optimisation process (and I guess, $D_{KL}(q_{\phi}||p_{\theta})$ decreases also, but sort of as a by-product ?). So does evidence change or not during the training process ?

Side-question : I think I'm getting confused about some very basic statistical notations as they are sometimes inconsistent between different materials. Is it true that usually, a notation such as $p(X)$ is a distribution, and $p(x)$ is a number representing the probability $p(X=x)$ ? So evidence is a number, right, not a distribution ?

$\endgroup$

2 Answers 2

4
$\begingroup$

The marginal log likelihood $\log p_\theta(x)$ doesn't depend on $q_\phi$: the variational distribution $q_\phi$ is introduced only to allow a numerically tractable decomposition of $\log p_\theta(x)$ (put it differently, $q_\phi$ cancels out from the right hand side of your first equation). So any change to the variational parameters $\phi$ will not modify the $\log p_\theta(x)$.

However the evidence lower bound (ELBO) does depend on $q_\phi$—changes in $\phi$ will affect the ELBO.

Maybe it's easier to understand what happens if we consider the alternating optimization procedure (as it is the case with expectation maximization).

  1. If we keep the parameters $\theta$ fixed and maximize the ELBO with respect to $\phi$ that reduces the gap to the true marginal log likelihood (ideally, reaching $\log p_\theta(x)$). Note that this step doesn't change the log likelihood.

  2. If we keep the variational parameters $\phi$ fixed and maxize the ELBO with respect to $\theta$ that increases the lower bound and as a consequence also the marginal log likelihood (provided that the ELBO was tight already).

For completeness, let me include Figures 9.12 and 9.13 from Bishop's Pattern Recognition and Machine Learning; they might offer a helpful visual depiction of the underlying process:

enter image description here enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for this answer ! Considering the optimization procedure as a two-step procedure similar to what EM does really pushed my understanding forward. Digging deeper in the link between VAEs and EM, I found this answer stats.stackexchange.com/a/479618/201218 to also be very informative. $\endgroup$
    – Soltius
    Jan 26 at 10:11
1
$\begingroup$

In the context of VAE, you have two moving parts - $q_\phi(z|x)$ and $p_\theta(x|z)$. You only increase the evidence by optimizing the $\theta$'s (the weights of the decoder) - basically doing maximum likelihood. You do this without any regard to VI, or to $\phi$. You then find the encoder by doing VI, and optimizing the encoder weights.

So, does evidence change or not during the training process ?

It does, but it's done regardless of VI. So, the above video explanation does not relate to it at all.

The evidence is a function that depends on the data. For a given dataset it will be a normalizing constant / number.

$\endgroup$
2
  • $\begingroup$ Thanks for the answer ! Really helped clear things up that the VAE training actually optimizes two different part, and VI only relates to one of them. Indeed, the video I linked to only relates to VI, and so in this context, evidence is constant (as theta is never optimized). $\endgroup$
    – Soltius
    Jan 26 at 10:13
  • $\begingroup$ @Soltius You're welcome. I just finished creating a YouTube video about VAEs - you might find it useful: youtube.com/watch?v=vJo7hiMxbQ8 $\endgroup$ Jan 26 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.