# Comparing two PDFs using Kullback-Leibler divergence

I am trying to compare two PDFs using Kullback-Leibler divergence but I am getting a value which means they are almost identical. Am I missing something? Here is my code.

import numpy
import scipy.stats as ss
import matplotlib.pyplot as plt

a = numpy.random.rand(1000)
b = numpy.random.rand(650)
ag = ss.gaussian_kde(a)
bg = ss.gaussian_kde(b)
bk = numpy.linspace(numpy.min(b), numpy.max(b), 1000)
ak = numpy.linspace(numpy.min(a), numpy.max(a), 1000)
agv = ag(ak)
bgv = bg(bk)
e = ss.entropy(agv, bgv)
0.008704001913773865

plt.plot(ak,agv)
plt.show()


plt.plot(bk,bgv)
plt.show()


These two seems not equal. But why am I getting such a low entropy?

• What does ss.entropy do? I'm not familiar with Python. In R I would integrate numerically density(a)*log(density(a)/density(b)), in 0,1. – utobi Nov 12 '16 at 6:45
• – Math1000 Nov 12 '16 at 8:51
• Seems to do the right thing. However, although the KL value here may seem pretty small, it is not easy to say "how much small is small" ... – utobi Nov 12 '16 at 11:29
• Is it too small? I mean both $a$ and $b$ are being sampled from uniform(0,1) – sntx Apr 11 '17 at 17:44