I'm trying to fully understand Variational Autoencoders (VAEs) 1 and their math but one part keeps confusing me and I hope someone can give me in an intuitive explanation what I am missing.

Here is how I understood it and where I am getting confused:

The overall loss consists of 2 parts, the regularization loss and the reconstruction loss:


The regularization loss ensures that our VAE encoder $q(z|x)$ produces latents that follow our prior distribution $p(z)$ (gaussian normal): $$L_{regularization} = KL(q(z|x) \parallel p(z))$$ if $q(z|x)$ is close to $p(z)$, the KL term is close to 0, hence, our goal is to minimize this term, because we want our latents to follow that prior distribution.

The reconstruction loss is computed based on maximizing the log-likelihood of our output given the latent variable z: $p(x|z)$. In other words, we try make sure that our output looks like our input, i.e. "reconstructing" it from our latents. And since maximizing the log-likelihood is the same as minimizing the negative log-likelihood, we can write:

$$L_{reconstruction} =-\mathbb{E}_{q(z|x)}[logp(x|z)]$$

From this follows:

$$L_{vae}= KL(q(z|x) \parallel p(z)) -\mathbb{E}_{q(z|x)}[logp(x|z)]$$

Now here is where I am confused:

In Kingma & Welling, "Auto-Encoding Variational Bayes" (2014) and everywhere else I am looking, the final loss is the above loss $*(-1)$:

$$L_{vae}= -KL(q(z|x) \parallel p(z)) +\mathbb{E}_{q(z|x)}[logp(x|z)]$$ or $$L_{vae}= \mathbb{E}_{q(z|x)}[logp(x|z)]-KL(q(z|x) \parallel p(z))$$

Main Question: Usually we do such a sign switch, since we want to convert our problem from a maximization to a minimization problem for optimizing it (e.g. with a neural network). But why are we doing it here? Are we not already "minimizing the negative log-likelihood" and is the KL term not already a minimization problem?


1 Answer 1


The question is starts from a mistaken premise. Equations $(1), (2), (3)$ in the article (Kingma & Welling, "Auto-Encoding Variational Bayes," 2014), refer to the likelihood, not the loss function. Likelihoods are maximized; maximizing $f(x)$ is equivalent to minimizing $-f(x)$. This quotation from the paper shows that these equations are likelihoods.

The marginal likelihood is composed of a sum over the marginal likelihoods of individual datapoints $\log p_\theta(x^{(1)}, \cdots, x^{(N)}) = \sum_{i=1}^N \log p_\theta(x^{(i)})$, which can each be rewritten as: \begin{align} \log p_\theta(x^{(i)}) = D_{KL}(q_\phi(z|x^{(i)})||p_\theta(z|x^{(i)})) + \mathcal{L}(\theta,\phi;x^{(i)}) \tag{1}\label{1} \end{align} The first RHS term is the KL divergence of the approximate from the true posterior. Since this KL-divergence is non-negative, the second RHS term $\mathcal{L}(\theta,\phi;x^{(i)})$ is called the (variational) lower bound on the marginal likelihood of datapoint $i$, and can be written as: \begin{align} \log p_\theta(x^{(i)}) \geq \mathcal{L}(\theta,\phi;x^{(i)}) &= \mathbb{E}_{q_\phi(z|x)}[- \log q_\phi(z|x) + \log p_\theta(x,z)] \tag{2}\label{2} \end{align} which can also be written as: \begin{align} \mathcal{L}(\theta,\phi;x^{(i)}) = - D_{KL}(q_\phi(z|x^{(i)}) || p_\theta(z)) + \mathbb{E}_{q_\phi(z|x^{(i)})} {\log p_\theta(x^{(i)} | z)} \tag{3}\label{3} \end{align}

(I've bolded the word “likelihood” in this quotation.)

  • $\begingroup$ I am still struggle to understand get an intuitive understanding of the two loss components. I get that we want to maximize the log-likelihood term (reconstruction loss), but why are we maximizing the KL-term (regularization loss)? Do we not want the KL to be as close to 0 as possible, i.e. minimize it? How can we maximize both a maximization and a minimization term at the same time? or does "maximize the KL term" mean "maximizing a minimization problem means we are maximizing it, if we the term is close to 0"? $\endgroup$
    – SvenG
    Jun 5 at 9:39
  • $\begingroup$ In equation 3, which is a likelihood, and therefore is maximized, KLD has a negative sign. Or, if you like, write down equation (3) with reversed sign and recall that it is a loss, so it is minimized. $\endgroup$
    – Sycorax
    Jun 5 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.