Why is KL divergence between a standard normal, and normal distribution 0 if standard deviation is 0.37? I wanted to see a graph about how KL divergence between a standard normal distribution, and any other normal distributions with 0 mean, and standard deviation being $x$ varies.
I mostly need it for variational autoencoder loss calculation, and so far I learnt that it needs to be calculated like:
$$D_{KL} = 0.5 * (\sigma^2 + \mu^2 - 1 - log \space \sigma^2)$$
If we pretend we know that $\mu$ is 0, I'm assuming we can simply skip that step, so we end up with:
$$D_{KL} = 0.5 * (\sigma^2 - 1 - log \space \sigma^2)$$
In this graph you can see that the KL divergence is 0 if $x$ is -1, 1, -0.37, 0.37. If my equation is right, it should mean how different a normal distribution is with $x$ standard deviation from a standard normal distribution. I don't understand the negative values, but certainly don't understand the 0.37 value.
Did I mess up my equation, or it is expected?
 A: The formula is correct. Just recall that KL divergence is defined in terms of the natural logarithm.
The software you used to make the plot appears to be using the convention $\log=\log_{10}$ . If you apply the change of base formula, or use $\ln$,  then you get the correct graph. So either

*

*$-0.5\left(1 + \frac{2 \log_{10}(x)}{\log_{10} e}-x^2\right)$ or

*$-0.5\left(1 + 2 \ln(x)-x^2\right)$ works correctly.

The lesson here is to know what convention your software is using, because it can be inconsistent between software. If you plot it in  Wolfram Alpha for instance, then the convention $\log = \log_e$ is used.
https://www.wolframalpha.com/input/?i=-0.5*%281%2B+2+log%28x%29+-+x%5E2%29
A: KL divergence between two normal RVs is $$D_{KL}(\mathcal N_1,\mathcal N_2)=\log {\frac{\sigma_2}{\sigma_1}}+\frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2}-{1\over 2}$$
KL divergence is not symmetric, so if you take $\mu_2=0, \sigma_2=1$, the expression reduces to
$$D_{KL}=-\log\sigma_1+\frac{\sigma_1^2+\mu_1^2}{2}-{1\over 2}$$
If you take the opposite, i.e. $\mu_1=0, \sigma_1=1$:
$$D_{KL}=\log\sigma_2+\frac{1+\mu_2^2}{2\sigma^2}-{1\over 2}$$
The first one is in effect the same as yours. As @Sycorax (+1) pointed out if you just use $0.5\cdot\left(x^{2}-1-\ln x^{2}\right)$ instead of $0.5\cdot\left(x^{2}-1-\log x^{2}\right)$ in the plotting service.
