3
$\begingroup$

I've been trying to get the KL divergence for two lognormal distributions. I know what it is for the univariate case,

$$ D(f_i\|f_j)= \frac1{2\sigma_j^2}\left[(\mu_i-\mu_j)^2+\sigma_i^2-\sigma_j^2\right] + \ln \frac{\sigma_j}{\sigma_i}, $$

however I've not been able to find anything on the multivariate case.

$\endgroup$
0

1 Answer 1

1
$\begingroup$

Assume $p(x)= \log\mathcal{N}(x|\mu_p,\Sigma_p)$ and $q(x)= \log\mathcal{N}(x|\mu_q,\Sigma_q)$. Where $x\in R^D$ and $D$ is the dimension of $x$. So $x = [x_1,x_2, \ldots ,x_D]^T.$

Based on KL-divergence definition we have: $$D_{KL}(p(x)||q(x))=\int\limits_{R^D} p(x)\log\dfrac{p(x)}{q(x)}=\int\limits_{R^D}p(x)\log (p(x)) - \int\limits_{R^D}p(x)\log(q(x))=E_p\{log(p(x))\} - E_p\{log(q(x))\}.$$

Before taking the first expectation, let's first determine the $\log(p(x))$: $$log(p(x))=-\frac{D}{2}\log(2\pi)-0.5\log(|\Sigma_p|)-\sum\limits_{i=1}^Dlog(x_i) - 0.5(\log(x)-\mu_p)^T\Sigma_p^{-1}(\log(x)-\mu_p).$$

The important thing to remember is that if for example $z\sim\log\mathcal{N}(\mu,\Sigma)$ then $\log(z)\sim\mathcal{N}(\mu,\Sigma)$.

Now let's take the expectation of $\log(p(x))$: $$E_p\{log(p(x))\}=-\frac{D}{2}\log(2\pi)-0.5\log(|\Sigma_p|)-\sum\limits_{i=1}^D\mu_p(i) - 0.5*trace(I_D).$$

Similarly for $E_p\{log(q(x))\}$: $$E_p\{log(q(x))\}=-\frac{D}{2}\log(2\pi)-0.5\log(|\Sigma_q|)-\sum\limits_{i=1}^D\mu_p(i)-0.5*trace(\Sigma_p\Sigma_q^{-1})-0.5(\mu_p-\mu_q)^T\Sigma_q^{-1}(\mu_p-\mu_q).$$

So for $D_{KL}(p||q)$ :

$$D_{KL}(p||q)=0.5\log(\dfrac{|\Sigma_q|}{|\Sigma_p}|)-D/2 + 0.5 trace(\Sigma_p\Sigma_q^{-1})+0.5(\mu_p-\mu_q)^T\Sigma_q^{-1}(\mu_p-\mu_q).$$

you can see that the KL divergence between two lognormals is equal to the Kl divergence between two normal distributions with the same parameter $\mu$ and $\Sigma$.

$\endgroup$
4
  • $\begingroup$ There is an error here. If z distributed log normal then it is exp(z) that is normal and not log(z). $\endgroup$ Dec 28, 2017 at 2:12
  • $\begingroup$ @Michael Chernick I haven't checked the correctness of the answer, but you are the one who's mixed up. The log of a lognormal is normal. The exp of a normal is lognormal. $\endgroup$ Dec 28, 2017 at 2:15
  • $\begingroup$ Yes log of lognormal is normal. I skipped some of the derivation in the answer because after taking the log, the derivation is similar to KL-divergence between two Gaussian. $\endgroup$
    – nomadr
    Dec 30, 2017 at 1:17
  • $\begingroup$ Is there some crazy coincidence here, or is KL divergence transformation invariant in some interesting way? $\endgroup$ Aug 7, 2022 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.