Which exact loss do we minimize in a VAE model? Reading about VAEs here and there, I often get stuck in the confusion about which quantity gets minimized as VAE objective.
After some calculations, here's what we get at:
$\log p_\theta(x) \ge - \mathbb{E}_{q_\phi}[\log \, q_\phi(z|x)] + \mathbb{E}_{q_\phi}[\log \, p_{\theta}(z,x)] $
The component on the right is what is commonly known as ELBO. It can be rewritten as:
$$ELBO(\phi) = \mathbb{E}_{q_\phi}[\log \, p_\theta(x|z)] - \mathbb{E}_{q_\phi} \left[\log \, \frac{q_\phi (z|x)}{p_\theta(z)}\right]$$
This quantity is maximixed (from what I've understood). Does this mean that in the concrete neural network implementations we need to provde $-ELBO(\phi)$ as loss function?
note: for simplicity i use $\theta$ interchangebly above even though they might be different.
 A: Yes, maximizing the ELBO is equivalent to minimizing the negative ELBO. This is a sign convention. You minimize the negative ELBO (also called the variational free energy) in the standard training objective for a variational autoencoder.
A: Yes!
As you stated correctly in the inequality, the ELBO is a lower bound of the evidence (log(p(x))). The difference between the evidence and the ELBO is the KL divergence between the true posterior p(z|x) and the approximate posterior q(z|x).
Since our objective is to minimize the ELBO (to bring our approximate posterior as close as possible to the true one) and the evidence is independent of q (correctly noted independent of the parameters of q, if we are taking a parametric approximation for q), minimizing this KL divergence leads to maximizing the ELBO. Because we can not deal with the KL and its optimization, we use the ELBO instead. In order to use the popular libs to optimize our objective (max the ELBO), we need to take -ELBO and minimize it instead.
I hope this helps!
