3
$\begingroup$

I wanted to see a graph about how KL divergence between a standard normal distribution, and any other normal distributions with 0 mean, and standard deviation being $x$ varies.

I mostly need it for variational autoencoder loss calculation, and so far I learnt that it needs to be calculated like:

$$D_{KL} = 0.5 * (\sigma^2 + \mu^2 - 1 - log \space \sigma^2)$$

If we pretend we know that $\mu$ is 0, I'm assuming we can simply skip that step, so we end up with:

$$D_{KL} = 0.5 * (\sigma^2 - 1 - log \space \sigma^2)$$

In this graph you can see that the KL divergence is 0 if $x$ is -1, 1, -0.37, 0.37. If my equation is right, it should mean how different a normal distribution is with $x$ standard deviation from a standard normal distribution. I don't understand the negative values, but certainly don't understand the 0.37 value.

Did I mess up my equation, or it is expected?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ It is always non-negative, so you messed up somewhere. With $\mu=0$, it should be $\frac{1}{2}(\sigma^{-2}-1)-\log{\sigma}$. As a function of $\sigma$, the derivative is $\sigma-\frac{1}{\sigma}$ and therefore the divergence is decreasing on $(0,1)$, then increasing on $(1,\infty)$. $\endgroup$
    – John L
    Commented Feb 24, 2021 at 20:46
  • $\begingroup$ KL divergence between p and q is maximised at 0, which occurs only when p and q are the same (almost everywhere) $\endgroup$
    – innisfree
    Commented Feb 25, 2021 at 12:54

2 Answers 2

Reset to default
4
$\begingroup$

The formula is correct. Just recall that KL divergence is defined in terms of the natural logarithm.

The software you used to make the plot appears to be using the convention $\log=\log_{10}$ . If you apply the change of base formula, or use $\ln$, then you get the correct graph. So either

  • $-0.5\left(1 + \frac{2 \log_{10}(x)}{\log_{10} e}-x^2\right)$ or
  • $-0.5\left(1 + 2 \ln(x)-x^2\right)$ works correctly.

The lesson here is to know what convention your software is using, because it can be inconsistent between software. If you plot it in Wolfram Alpha for instance, then the convention $\log = \log_e$ is used.

https://www.wolframalpha.com/input/?i=-0.5*%281%2B+2+log%28x%29+-+x%5E2%29

Improve this answer
$\endgroup$
2
$\begingroup$

KL divergence between two normal RVs is $$D_{KL}(\mathcal N_1,\mathcal N_2)=\log {\frac{\sigma_2}{\sigma_1}}+\frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2}-{1\over 2}$$

KL divergence is not symmetric, so if you take $\mu_2=0, \sigma_2=1$, the expression reduces to $$D_{KL}=-\log\sigma_1+\frac{\sigma_1^2+\mu_1^2}{2}-{1\over 2}$$

If you take the opposite, i.e. $\mu_1=0, \sigma_1=1$: $$D_{KL}=\log\sigma_2+\frac{1+\mu_2^2}{2\sigma^2}-{1\over 2}$$

The first one is in effect the same as yours. As @Sycorax (+1) pointed out if you just use $0.5\cdot\left(x^{2}-1-\ln x^{2}\right)$ instead of $0.5\cdot\left(x^{2}-1-\log x^{2}\right)$ in the plotting service.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I've seen many implementations of VAE's KL loss function, and seen a bunch of different variations. Checked, and most of the examples use the equation as above (updated the question). If the above equation is right, and assuming we know that $\mu_1$, and $\mu_2$ are both 0, we can simply skip the $ + \mu^2$ part, so we left with $0.5 * (\sigma^2 - 1 - log \space \sigma^2)$. But the graph is still the butt like shape, as of the link above to it. $\endgroup$
    – Gergő Horváth
    Commented Feb 25, 2021 at 12:13
  • $\begingroup$ @gunes You're missing a constant. KLD is minimized at 0 when the distributions are the same, but your equations do not achieve 0 when they have the same parameters. $\endgroup$
    – Sycorax ♦
    Commented Feb 25, 2021 at 13:25
  • $\begingroup$ @Sycorax thanks for the heads up, I corrected it. $\endgroup$
    – gunes
    Commented Feb 25, 2021 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.