0
$\begingroup$

This question already has an answer here:

I am trying to re-create a variational autoencoder. The loss function has two terms: reconstruction loss and KL-divergence term. KL-divergence is defined as $$ D_{KL}(P||Q) = -\sum_{x\in X}{P(X)\log\bigg(\frac{Q(X)}{P(X)}}\bigg)$$

While in the the code here it says

kl_loss = - 0.5 * K.mean(1 + z_log_sigma - K.square(z_mean) - K.exp(z_log_sigma), axis=-1)

The formula for KL-divergence looks completely different from what is in this line of code. Can somebody familiar with variational autoencoders help?

$\endgroup$

marked as duplicate by kjetil b halvorsen, mdewey, mkt, Siong Thye Goh, tomka May 10 at 13:40

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

migrated from math.stackexchange.com May 8 at 3:29

This question came from our site for people studying math at any level and professionals in related fields.

2
$\begingroup$

Note that this implementation assumes Gaussian distributions, so this loss (as far as I understand it, it would be helpful if yo uactually used the conventional notation used in math instead of some code snippet) measures the KL divergence between gaussians. You can find the whole derivation here on stats.se:

https://stats.stackexchange.com/questions/7440/kl-divergence-between-two-univariate-gaussians

$\endgroup$
2
$\begingroup$

It'll help to first rewrite the code in MathJax, viz. $$D_{KL}^\text{other}(P\Vert Q)=-\frac12\left(1+\ln\sigma^2-\bar{z}^2-\sigma^2\right).$$(Unfortunately, whoever wrote the code gave the false impression in the naming that z_log_sigma denotes $\ln\sigma$ rather than $\ln\sigma^2$.) With $\bar{z}=\mu_1-\mu_2$ this formula becomes$$D_{KL}^\text{other}(P\Vert Q)=-\frac12-\ln\sigma+\frac12(\mu_1-\mu_2)^2+\frac12\sigma^2.$$This is the $\sigma_1=\sigma,\,\sigma_2=1$ special case of the two-Gaussians KL divergence in @flawr's link.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.