I was studying VAEs and came accross the Loss function that consists of KL Divergence. I wanted to intuitively make sense of the KL divergence part of the loss function. It would be great if somebody can help me. Thanks. enter image description here


The KL divergence tells us how well the probability distribution Q approximates the probability distribution P by calculating the cross-entropy minus the entropy. Intuitively, you can think of that as the statistical measure of how one distribution differs from another.

In VAE, let $X$ be the data we want to model, $z$ be latent variable, $P(X)$ be the probability distribution of data, $P(z)$ be the probability distribution of the latent variable and $P(X|z)$ be the distribution of generating data given latent variable

In the case of variational autoencoders, our objective is to infer $P(z)$ from $P(z|X)$. $P(z|X)$ is the probability distribution that projects our data into latent space. But since we do not have the distribution $P(z|X)$, we estimate it using its simpler estimation $Q$.

Now while training our VAE, the encoder should try to learn the simpler distribution $Q(z|X)$ such that it is as close as possible to the actual distribution $P(z|X)$. This is where we use KL divergence as a measure of a difference between two probability distributions. The VAE objective function thus includes this KL divergence term that needs to be minimized.

$$ D_{KL}[Q(z|X)||P(z|X)] = E[\log {Q(z|X)} − \log {P(z|X)}] $$

  • $\begingroup$ Thanks for the response, But I actually wanted to know that how did we reach from DKL[Q(z|X)||P(z|X)]=E[logQ(z|X)−logP(z|X)] to the equation mentioned in my question $\endgroup$ – quantumbiker Mar 1 at 8:33
  • $\begingroup$ You may find the full derivation here. $\endgroup$ – Krishn Bera Mar 1 at 8:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.