I'm studying variational autoencoders and I cannot get my head around their cost function.

I understood the principle intuitively but not the math behind it: in the paragraph 'Cost Function' of the blog post here it is said:

In other words, we want to simultaneously tune these complementary parameters such that we maximize log(p(x|ϕ,θ)) - the log-likelihood across all datapoints xx under the current model settings, after marginalizing out the latent variables zz. This term is also known as the model evidence.

in other words we want to solve the log-likelihood function from the encoder/decoder parameters ϕ and θ in order to find the probability distribution p from the input samples that better models them.

After that

We can express this marginal likelihood as the sum of what we’ll call the variational or evidence lower bound LL and the Kullback-Leibler (KL) divergence DKLDKL between the approximate and true latent posteriors: log(p(x))=L(ϕ,θ;x)+DKL(qϕ(z|x)||pθ(z|x))

This is also just math from the definition of KL divergence as in the passages here.

for now the important thing is that it is non-negative by definition; consequently, the first term acts as a lower bound on the total. So, we maximize the lower bound LL as a (computationally-tractable) proxy for the total marginal likelihood of the data under the model.

This is also reasonable to me.

With some mathematical wrangling, we can decompose LL into the following objective function:

enter image description here

I tried to follow the 'mathematical wrangling' on the paper linked (2.2 - The variational bound) but couldn't obtain the function above.

Can somebody help me out in figuring it?


2 Answers 2


To get the variational objective, begin with marginal likelihood:

$p(x) = \int p(x, z) dz = \int \frac{q(z)}{q(z)} p(x,z) dz$

Recognise this as an expectation:

$ = \mathbb E_q [\frac{p(x,z)}{q(z)}] $

Use Jensen's inequality

$\ln p(x) \geq \mathbb E_q [\ln \frac{p(x,z)}{q(z)}] $

Use def of conditional probability and properties of logs:

$ = \mathbb E_q [\ln p(x|z) + \ln p(z) - \ln q(z)] $

Use def of KL divergence:

$ = \mathbb E_q [\ln p(x|z)] - KL(q||p)$

Sprinkle in some $\theta$ and $\phi$ as appropriate and we have the format required. The conditioning of $z$ on $x$ in their notation $q_\phi(z|x)$ is a little unnecessary & confusing IMHO since $q$ is selected to minimise divergence from the posterior (although I get they're flagging it's an approx to the posterior rather than the prior).

  • $\begingroup$ "$q$ is selected to minimise the divergence from the posterior" is certainly true for something like mean-field variational inference, but is this also true for amortised forms of variational inference such as auto-encoding VB, where a neural neural network selects $q$ based on $x$? I don't see how you could drop the dependence on $x$ in that case. ($q(z \mid x)$ is still not great notation since it's not really a conditional distribution – $q(z, x)$ is not defined. $q(z; x)$ may be more appropriate.) $\endgroup$
    – Lucas
    Commented Sep 21, 2017 at 15:34
  • $\begingroup$ Hmm I just think talking about $q(z|x)$ implies $q(z)$ is a thing, like prior and posterior, when really there is only one type of $q$ function. Gwet where you're coming from though, $\endgroup$ Commented Sep 22, 2017 at 9:26

Although the answer above is totally correct, you can reach the same conclusion by playing around with the KL divergence. See my detailed answer with some nice references on VI.

As a side note, the joint $q(x,z)$ might not be defined but they use $q(z|x)$ to imply the dependence on the data. The example they choose in their original paper (Gaussian multivariate with the mean $\mu$ depending on $x$) might make a bit more sense on why they use the notation they use.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.