Is there a closed-form formula for the following KL divergence?
$D_{KL}(X,Y)$
where
$X \sim \mathrm{Gamma}(k,\theta)$
and
$Y \sim \mathrm{LogNormal}(\mu,\sigma^2)$
$\begingroup$
$\endgroup$
1
-
$\begingroup$ A lot of such results can be found here: mast.queensu.ca/~linder/pdf/GiAlLi13.pdf $\endgroup$– kjetil b halvorsen ♦Nov 30, 2017 at 12:35
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
11
Given: Let our $\text{Gamma}(k,\theta)$ random variable have pdf $f(x)$:
and let our $\text{Lognormal}(\mu, \sigma)$ random variable have pdf $g(x)$:
Then, the Kullback-Leibler divergence between the true distribution $f$ and the Lognormal approximation $g$ is given by:
$$E_f\big[\log f(x)\big] - E_f\big[\log g(x)\big]$$
The first term $E_f\big[\log f(x)\big]$ is:
and the second term $E_f\big[\log g(x)\big]$ is:
The solution is $P-Q$.
Notes:
- The
Expectfunction is from the mathStatica package for Mathematica. PolyGamma[n,z]denotes the $n^{th}$ derivative of the digamma function $\psi(z)=\frac{\Gamma'(z)}{\Gamma(z)}$
-
2$\begingroup$ It is not a numeric approximation. As to the definition of what a closed-form is, ... that is always open for discussion, ... but I think most would agree that the log and digamma functions constitute closed forms. $\endgroup$– wolfiesApr 17, 2015 at 4:12
-
1$\begingroup$ +1 I didn't have to read past the second line to know whose answer this was. $\endgroup$– Glen_bApr 17, 2015 at 4:40
-
1$\begingroup$ @AleksandrBlekh It should be trivially true that KL divergence exists for other distributions: for example, KL divergence with the uniform distribution becomes the entropy. Also, I don't actually think it's true that a closed form exists for mixtures of Gaissians. $\endgroup$– DanicaApr 17, 2015 at 4:50
-
1$\begingroup$ @AleksandrBlekh Yes, I also meant the closed form and should have said that. My example was that the divergence from a uniform is the entropy of the first distribution, so any distribution with a closed-form entropy has at least one set of closed-form KL divergences. And it's important to note that the closed-form JR divergence for GMMs only exists for parameter alpha=2 (the quadratic entropy), not for all JR divergences. (Indeed, the KL is a limit case of the JR divergence as alpha goes to 1.) $\endgroup$– DanicaApr 17, 2015 at 5:42
-
1$\begingroup$ @Dougal: Thank you for clarification - I see this now, browsing the paper again. BTW, if you're interested, I ran across an interesting paper, discussing and specifying sufficient conditions for a probability distribution family for a mixture of two of such distributions to have a closed form for some non-KL divergence measures, including JR. $\endgroup$ Apr 17, 2015 at 5:50