# KL divergence between a gamma distribution and a lognormal distribution?

Is there a closed-form formula for the following KL divergence?

$D_{KL}(X,Y)$

where

$X \sim \mathrm{Gamma}(k,\theta)$

and

$Y \sim \mathrm{LogNormal}(\mu,\sigma^2)$

Given: Let our $\text{Gamma}(k,\theta)$ random variable have pdf $f(x)$:

and let our $\text{Lognormal}(\mu, \sigma)$ random variable have pdf $g(x)$:

Then, the Kullback-Leibler divergence between the true distribution $f$ and the Lognormal approximation $g$ is given by:

$$E_f\big[\log f(x)\big] - E_f\big[\log g(x)\big]$$

The first term $E_f\big[\log f(x)\big]$ is:

and the second term $E_f\big[\log g(x)\big]$ is:

The solution is $P-Q$.

Notes:

1. The Expect function is from the mathStatica package for Mathematica.
2. PolyGamma[n,z] denotes the $n^{th}$ derivative of the digamma function $\psi(z)=\frac{\Gamma'(z)}{\Gamma(z)}$
• It is not a numeric approximation. As to the definition of what a closed-form is, ... that is always open for discussion, ... but I think most would agree that the log and digamma functions constitute closed forms. Commented Apr 17, 2015 at 4:12
• +1 I didn't have to read past the second line to know whose answer this was. Commented Apr 17, 2015 at 4:40
• @AleksandrBlekh It should be trivially true that KL divergence exists for other distributions: for example, KL divergence with the uniform distribution becomes the entropy. Also, I don't actually think it's true that a closed form exists for mixtures of Gaissians. Commented Apr 17, 2015 at 4:50
• @AleksandrBlekh Yes, I also meant the closed form and should have said that. My example was that the divergence from a uniform is the entropy of the first distribution, so any distribution with a closed-form entropy has at least one set of closed-form KL divergences. And it's important to note that the closed-form JR divergence for GMMs only exists for parameter alpha=2 (the quadratic entropy), not for all JR divergences. (Indeed, the KL is a limit case of the JR divergence as alpha goes to 1.) Commented Apr 17, 2015 at 5:42
• @Dougal: Thank you for clarification - I see this now, browsing the paper again. BTW, if you're interested, I ran across an interesting paper, discussing and specifying sufficient conditions for a probability distribution family for a mixture of two of such distributions to have a closed form for some non-KL divergence measures, including JR. Commented Apr 17, 2015 at 5:50