I am trying to compare two PDFs using Kullback-Leibler divergence but I am getting a value which means they are almost identical. Am I missing something? Here is my code.

import numpy
import scipy.stats as ss
import matplotlib.pyplot as plt

a = numpy.random.rand(1000)
b = numpy.random.rand(650)
ag = ss.gaussian_kde(a)
bg = ss.gaussian_kde(b)
bk = numpy.linspace(numpy.min(b), numpy.max(b), 1000)
ak = numpy.linspace(numpy.min(a), numpy.max(a), 1000)
agv = ag(ak)
bgv = bg(bk)
e = ss.entropy(agv, bgv)


a - PDF


b - PDF

These two seems not equal. But why am I getting such a low entropy?

  • 1
    $\begingroup$ What does ss.entropy do? I'm not familiar with Python. In R I would integrate numerically density(a)*log(density(a)/density(b)), in 0,1. $\endgroup$ – utobi Nov 12 '16 at 6:45
  • $\begingroup$ @utobi docs.scipy.org/doc/scipy/reference/generated/… $\endgroup$ – Math1000 Nov 12 '16 at 8:51
  • $\begingroup$ Seems to do the right thing. However, although the KL value here may seem pretty small, it is not easy to say "how much small is small" ... $\endgroup$ – utobi Nov 12 '16 at 11:29
  • $\begingroup$ Is it too small? I mean both $a$ and $b$ are being sampled from uniform(0,1) $\endgroup$ – sntx Apr 11 '17 at 17:44

That's the problem with divergences in general. As there isn't a scale for them it's not easy to say it's value is small.

What I use to do is computing KL for two PDFs I know should be close. Results lower than this indicate the two PDFs are close. This can be understood as a "calibration" for the measure.

Be aware that bigger the sample size, lower the divergence value. For instance, see how the color bar changes in the figures below (code based on yours: https://github.com/luizfrias/ica/blob/master/KL%20divergence%20test.ipynb)

enter image description here

enter image description here

| cite | improve this answer | |
  • 1
    $\begingroup$ In information theory, KL divergence does have a scale of bits. So if a and b were two probability distributions of byte values, to convert from a to b you'd need additionally to use 0.0087 bits /byte of data. And it is also bounded so that 0 < KL < 8 bits. $\endgroup$ – Paul Uszak Dec 19 '16 at 21:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.