Is there a closed-form formula for the following KL divergence?



$X \sim \mathrm{Gamma}(k,\theta)$


$Y \sim \mathrm{LogNormal}(\mu,\sigma^2)$


1 Answer 1


Given: Let our $\text{Gamma}(k,\theta)$ random variable have pdf $f(x)$:

enter image description here

and let our $\text{Lognormal}(\mu, \sigma)$ random variable have pdf $g(x)$:

enter image description here

Then, the Kullback-Leibler divergence between the true distribution $f$ and the Lognormal approximation $g$ is given by:

$$E_f\big[\log f(x)\big] - E_f\big[\log g(x)\big]$$

The first term $E_f\big[\log f(x)\big]$ is:

enter image description here

and the second term $E_f\big[\log g(x)\big]$ is:

enter image description here

The solution is $P-Q$.


  1. The Expect function is from the mathStatica package for Mathematica.
  2. PolyGamma[n,z] denotes the $n^{th}$ derivative of the digamma function $\psi(z)=\frac{\Gamma'(z)}{\Gamma(z)}$
  • 2
    $\begingroup$ It is not a numeric approximation. As to the definition of what a closed-form is, ... that is always open for discussion, ... but I think most would agree that the log and digamma functions constitute closed forms. $\endgroup$
    – wolfies
    Apr 17, 2015 at 4:12
  • 1
    $\begingroup$ +1 I didn't have to read past the second line to know whose answer this was. $\endgroup$
    – Glen_b
    Apr 17, 2015 at 4:40
  • 1
    $\begingroup$ @AleksandrBlekh It should be trivially true that KL divergence exists for other distributions: for example, KL divergence with the uniform distribution becomes the entropy. Also, I don't actually think it's true that a closed form exists for mixtures of Gaissians. $\endgroup$
    – Danica
    Apr 17, 2015 at 4:50
  • 1
    $\begingroup$ @AleksandrBlekh Yes, I also meant the closed form and should have said that. My example was that the divergence from a uniform is the entropy of the first distribution, so any distribution with a closed-form entropy has at least one set of closed-form KL divergences. And it's important to note that the closed-form JR divergence for GMMs only exists for parameter alpha=2 (the quadratic entropy), not for all JR divergences. (Indeed, the KL is a limit case of the JR divergence as alpha goes to 1.) $\endgroup$
    – Danica
    Apr 17, 2015 at 5:42
  • 1
    $\begingroup$ @Dougal: Thank you for clarification - I see this now, browsing the paper again. BTW, if you're interested, I ran across an interesting paper, discussing and specifying sufficient conditions for a probability distribution family for a mixture of two of such distributions to have a closed form for some non-KL divergence measures, including JR. $\endgroup$ Apr 17, 2015 at 5:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.