Variational Autoencoder - understanding the latent loss

Question

I'm studying variational autoencoders and I cannot get my head around their cost function.

I understood the principle intuitively but not the math behind it: in the paragraph 'Cost Function' of the blog post here it is said:

In other words, we want to simultaneously tune these complementary parameters such that we maximize log(p(x|ϕ,θ)) - the log-likelihood across all datapoints xx under the current model settings, after marginalizing out the latent variables zz. This term is also known as the model evidence.

in other words we want to solve the log-likelihood function from the encoder/decoder parameters ϕ and θ in order to find the probability distribution p from the input samples that better models them.

After that

We can express this marginal likelihood as the sum of what we’ll call the variational or evidence lower bound LL and the Kullback-Leibler (KL) divergence DKLDKL between the approximate and true latent posteriors: log(p(x))=L(ϕ,θ;x)+DKL(qϕ(z|x)||pθ(z|x))

This is also just math from the definition of KL divergence as in the passages here.

for now the important thing is that it is non-negative by definition; consequently, the first term acts as a lower bound on the total. So, we maximize the lower bound LL as a (computationally-tractable) proxy for the total marginal likelihood of the data under the model.

This is also reasonable to me.

With some mathematical wrangling, we can decompose LL into the following objective function:

I tried to follow the 'mathematical wrangling' on the paper linked (2.2 - The variational bound) but couldn't obtain the function above.

Can somebody help me out in figuring it?

Answer 1

To get the variational objective, begin with marginal likelihood:

$p(x) = \int p(x, z) dz = \int \frac{q(z)}{q(z)} p(x,z) dz$

Recognise this as an expectation:

$ = \mathbb E_q [\frac{p(x,z)}{q(z)}] $

Use Jensen's inequality

$\ln p(x) \geq \mathbb E_q [\ln \frac{p(x,z)}{q(z)}] $

Use def of conditional probability and properties of logs:

$ = \mathbb E_q [\ln p(x|z) + \ln p(z) - \ln q(z)] $

Use def of KL divergence:

$ = \mathbb E_q [\ln p(x|z)] - KL(q||p)$

Sprinkle in some $\theta$ and $\phi$ as appropriate and we have the format required. The conditioning of $z$ on $x$ in their notation $q_\phi(z|x)$ is a little unnecessary & confusing IMHO since $q$ is selected to minimise divergence from the posterior (although I get they're flagging it's an approx to the posterior rather than the prior).

Answer 2

Although the answer above is totally correct, you can reach the same conclusion by playing around with the KL divergence. See my detailed answer with some nice references on VI.

As a side note, the joint $q(x,z)$ might not be defined but they use $q(z|x)$ to imply the dependence on the data. The example they choose in their original paper (Gaussian multivariate with the mean $\mu$ depending on $x$) might make a bit more sense on why they use the notation they use.