# K-L divergence is 0 for clearly different distributions. Why?

I have two datasets whose densities look clearly different. However, when I use the KLdiv function, it gives me a value of 0 and I don't understand why. Any suggestions on where I am going wrong? Datasets are at the end of this post.

I did something like this:

library(flexmix)
KLdiv(cbind(density(A)$y, density(D)$y))


And it yields me:

     [,1] [,2]
[1,]    0    0
[2,]    0    0


JS divergence is zero too as a result of this:

0.5*(KLdiv(cbind(density(A)$y,0.5*(density(A)$y + density(D)$y)))) + 0.5*(KLdiv(cbind(density(D)$y,0.5*(density(A)$y + density(D)$y))))

[,1] [,2]
[1,]    0    0
[2,]    0    0


However, the pdfs of A and D are quite different:

Following are my two datasets:

A = c(53, 57, 54, 63, 66, 66, 75, 86, 77, 52494, 56722, 57097, 56754,
57399, 58145, 56582, 58429, 57868, 57349, 57569, 56507, 55925,
58177, 56455, 58162, 58253, 58340, 55172, 54767, 57750, 515,
510, 1686, 299, 571, 301, 300, 305, 300, 429, 975, 1364, 1377,
12122, 1255, 15845, 23767, 6883, 16844, 13867, 0, 0, 0, 8293,
6119, 7504, 6292, 6257, 6406, 1267601, 427055, 667541, 1209979,
760919, 1035907, 710933, 2217471, 725223, 888272, 1228833, 2153679,
1541844, 1310682, 1706112, 901743, 955793, 1900751, 54844, 2125783,
1109668, 2533008, 1476430, 3838432, 1594130, 889522, 884048,
1212327, 1407082, 2529584, 1410899, 1154139, 1216456, 1255570,
1550869, 2096869, 2360463, 2079124, 2239979, 5206460, 1350783,
2866784, 3487824, 610215, 2270840, 933909, 1049955, 901239, 1671539,
12292475, 2160519, 1730576, 1307651, 1659493, 1419815, 2333420,
2813600, 6260, 1557051, 2012477, 2383571, 1889906, 2497337, 3786140,
3233662, 4550030, 4765371, 4431086, 2143366, 1758742, 2520050,
2036522, 2945222, 4420937, 1452987, 4455005, 1838441, 1712123,
1213652, 1058007, 1156038, 11818454, 2913849, 5614698, 4188568,
4121445, 4067988, 5157146, 3858549, 3799965, 4465718, 5510926,
1426606, 4150290, 3995099, 5051489, 5518465, 6891987, 2599328,
2255332, 2921442, 2189249, 3552345, 4020428, 3752018, 4649134,
2982890, 3035131, 3337928, 7469273, 5306854, 3343986, 3690062,
4301378, 3280312, 3951362, 3074957, 3885109, 3965623, 3411739,
3275028, 2955563, 4281609, 3007306, 4932588, 1215474, 5038889,
3879138, 5686709, 99190451, 3857811, 21850538, 9272045, 23873337,
80095039, 99642585, 101780491, 60067705, 3546841, 2329745)

D = c(87983, 1130660, 721996, 881447, 0, 0, 56561, 148153, 82070,
88294, 81592, 303643, 1655103, 1660248, 1172794, 314841, 1585898,
339034, 464871, 1620502, 569710, 450522, 1152443, 1012775, 278482,
2019150, 8263736, 1913965, 1573202, 10456294, 13028222, 17432764,
19370911, 3683766, 12794272, 20638951, 14958438, 21847386, 33632736,
31652260, 23939674, 1432818, 37414993, 33164198, 34101979, 36387746,
41604477, 49173987, 39454209, 38942401, 44779660, 39630652, 37190054,
36059029, 45519990, 36309731, 44287159, 52460663, 45415958, 46605375,
70220155, 40907676, 0, 81266347, 71459813, 76837711, 112575185,
93436064, 109946521, 93868840, 122776950, 97212417, 108955821,
145587560, 216752084, 336859809, 185175516, 291129509, 349552593,
372988145, 292428463, 415094255, 185428625, 360299946, 387408971,
335578001, 393822458, 364598154, 416030537, 418917188, 448585165,
418281134, 380423194, 359740594, 510471342, 430134138, 463678777,
516374163, 512168586, 448045870, 491493753, 531045072, 499352572,
430560185, 507585264, 453550205, 436303557, 373408445, 169499536,
311790281, 468707162, 433377700, 335570055, 434688343, 358754832,
413128178, 358604964, 333417416, 103901186, 45295220, 107528512,
38698395, 68062269, 71930943, 63461445, 80691308, 84880325, 116910124,
78199074, 33932505, 98104155, 63883422, 2449889, 81687886, 112591823,
933357, 110882829, 167759933, 143331631, 128723052, 97515988,
89474519, 95248435, 77201398, 115732319, 150051651, 89815946,
119109327, 156889426, 104873286, 112453255, 103860299, 141042327,
161372922, 228847332, 279230413, 227022713, 245884032, 449327187,
399725637, 427264302, 385944417, 403338926, 384747124, 413700052,
373033651, 399978293, 455361833, 417924530, 473795493, 485446783,
551847677, 536782686, 551399627, 433065731, 434131355, 501258800,
468106521, 430447721, 408847961, 373866095, 360367708, 415444352,
439868563, 469183737, 572553168, 525927688, 417114578, 463202610,
335993664, 323369810, 319030248, 416164672, 393022155, 324094628,
324588328, 225682090, 218461312, 210500238, 204682022, 219859789,
244008146, 279700073, 231405316, 397901380, 351618973, 347947034,
475587874, 385887638, 441784135, 453616386, 425914531, 384140372,
394301503, 369180956, 359728515, 319432242, 340549134, 155250075,
390714221, 514657336, 357766885, 250607790, 313543688, 289122479,
289912982, 221289620, 212056585, 200354558, 172626254, 152561904,
141737182, 151121151, 135049494, 312876338, 155618873, 199563260,
243044037, 273332195, 302758331, 340552788, 340139019, 435174604,
467950978, 430091965, 345051079, 351816570, 263430776, 255793594,
164134822, 173371250, 254249143, 313999193, 148044484, 162452217,
130834412, 167723965, 200147738, 100376398, 103186023, 217542202,
302451572, 180647415, 262106015, 267346637, 186793712, 239398890,
312860449, 267037929, 159408789, 200059134, 132928248, 252577685,
255926249, 229046165, 113443872)

-
Any chance the numbers are too large? –  alex Jan 16 '13 at 20:06
I understand the frustration in not being able to comment so as to ask questions for additional relevant info (note that you will be able to when your rep = 50). However, please don't use the "Your Answer" feature to do this. (OTOH, if this is intended as a real answer, please edit it to flesh it out.) –  gung Jan 16 '13 at 20:45
@alex: I don't see why that should be a problem (at least with my limited knowledge). –  Legend Jan 16 '13 at 22:13
Alex is right -- KLdiv's docs says it truncates anything smaller than 1e-4 to 0; as you can easily check, both density(A)$y and density(D)$y fall below this threshold entirely, what explains zeros in output. Moreover, both densities have different x grid -- without equalising them with from and to arguments all results will make no sense. –  mbq Jan 16 '13 at 23:14
Can you please elaborate on without equalising them with from and to arguments? Do you mean, I should instead use: density(A, from=0, to=max_value)? –  Legend Jan 17 '13 at 2:22

I suspected numerical instability of some sort. What appears to be happening is that because the ranges of A and D are so large, the densities at each data point are very small. It appears as if KLdiv cuts off low densities at 1e-4 by default (this can be changed, but I don't know if you'll introduce problems that way). Also, I think you need to evaluate the two densities on the same grid.

-
I missed your edit... sorry for duplicating. –  mbq Jan 16 '13 at 23:15
+1 Thank You for your explanation. So, incorporating what mbq had suggested, i.e., computing P=density(A, from=0, to=max_value) and Q=density(D, from=0, to=max_value), can you suggest what I should pass on as arguments to KLdiv? Is it (P and Q) or (diff(P$x)*P$y and diff(Q$x)*Q$y)? –  Legend Jan 17 '13 at 2:25