Variational inference : is evidence constant?

Question

I'm studying variational inference (in the context of VAEs), and I'm watching this video at this time point. At this point in the video, the goal of approximating the intractable posterior $p_{\theta}(z|x)$ by an approximate posterior $q_{\phi}(z|x)$ has been set, and the ELBO formulation has been presented.

As is shown on the image, what the author of the video says at this point is that, since marginal log-likelihood will not change, maximizing ELBO minimizes the KL divergence between the $q$ and $p$ distributions (which is what we want). To quote more precisely, the author says : "We cannot compute [marginal log-likelihood] but we know that it will not change and so we have found a way around our KL divergence".

This sentence confused me. In the context of variational auto-encoders for generating data, I thought that the goal was specifically to increase the evidence by finding the optimal $\theta$ (and $\phi$) parameters, and so that evidence $p_{\theta}(x)$ will be maximized during the optimisation process (and I guess, $D_{KL}(q_{\phi}||p_{\theta})$ decreases also, but sort of as a by-product ?). So does evidence change or not during the training process ?

Side-question : I think I'm getting confused about some very basic statistical notations as they are sometimes inconsistent between different materials. Is it true that usually, a notation such as $p(X)$ is a distribution, and $p(x)$ is a number representing the probability $p(X=x)$ ? So evidence is a number, right, not a distribution ?

Answer 1

The marginal log likelihood $\log p_\theta(x)$ doesn't depend on $q_\phi$: the variational distribution $q_\phi$ is introduced only to allow a numerically tractable decomposition of $\log p_\theta(x)$ (put it differently, $q_\phi$ cancels out from the right hand side of your first equation). So any change to the variational parameters $\phi$ will not modify the $\log p_\theta(x)$.

However the evidence lower bound (ELBO) does depend on $q_\phi$—changes in $\phi$ will affect the ELBO.

Maybe it's easier to understand what happens if we consider the alternating optimization procedure (as it is the case with expectation maximization).

1. If we keep the parameters $\theta$ fixed and maximize the ELBO with respect to $\phi$ that reduces the gap to the true marginal log likelihood (ideally, reaching $\log p_\theta(x)$). Note that this step doesn't change the log likelihood.

2. If we keep the variational parameters $\phi$ fixed and maxize the ELBO with respect to $\theta$ that increases the lower bound and as a consequence also the marginal log likelihood (provided that the ELBO was tight already).

For completeness, let me include Figures 9.12 and 9.13 from Bishop's Pattern Recognition and Machine Learning; they might offer a helpful visual depiction of the underlying process:

Answer 2

In the context of VAE, you have two moving parts - $q_\phi(z|x)$ and $p_\theta(x|z)$. You only increase the evidence by optimizing the $\theta$'s (the weights of the decoder) - basically doing maximum likelihood. You do this without any regard to VI, or to $\phi$. You then find the encoder by doing VI, and optimizing the encoder weights.

So, does evidence change or not during the training process ?

It does, but it's done regardless of VI. So, the above video explanation does not relate to it at all.

The evidence is a function that depends on the data. For a given dataset it will be a normalizing constant / number.