# software library to compute KL divergence?

Are there any software libraries that compute KL divergences in closed form, that also give the derivatives of the KL divergence wrt the distributions' parameters? I'm using Julia, so it's particularly straightforward for me to call Julia, Fortran, C and C++ libraries.

Alternatively, if libraries like this don't exist, is there something I can do that would be easier than manually coding the KL divergences, and perhaps using automatic differentiation? I have to compute KL divergences for about 10 pairs of distribution with closed-form KL divergences, e.g. beta/beta, log-normal/log-normal, mv-normal/mv-normal, wrapped-cauchy/uniform.

• The package Distances.jl has code to evaluate KL-divergence for a particular sample. Apr 17, 2015 at 2:30
• I need it for distributions, not samples, unfortunately.
– Jeff
Apr 17, 2015 at 2:41
• I figured, thats why I posted it as a comment. Since Julia is a budding open source community, why not just write up the code yourself. If they are closed form it shouldn't be too difficult. It could make a good contribution to the Distributions.jl package. Apr 17, 2015 at 2:43
• As far as I understand, a closed form for KL divergence exists only for Gaussians or mixtures of Gaussians. Can you use numerical approximation? Apr 17, 2015 at 3:22
• No, they exist for many pairs of common distributions, e.g. beta/beta: [en.wikipedia.org/wiki/…
– Jeff
Apr 17, 2015 at 3:28

It's great that you came up with the solution (+1). I meant to post an answer to this question much earlier, but was busy traveling to my dissertation defense (which was successful :-). You are likely to be happy with your solution, but, in addition to possibility to compute KL divergences for certain distributions in R, for example, via function KLdiv from flexmix package (http://www.inside-r.org/packages/cran/flexmix/docs/KLdiv), I ran across another and, in my opinion, much better option, which might be of your interest.