# Is it Possible to Calculate Information Distances like the KL and Jensen-Shannon Divergences on EDFs and CDFs?

With the data I'm currently analyzing, it is easier to derive the EDFs than the PDFs; furthermore, I'm much more comfortable using CDF-EDF goodness-of-fit tests like the Kolmgorov-Smirnov and Anderson-Darling, which I already have code for. I would also like to calculate a multitude of information and probabilistic distances (also metrics and divergences; I'm not using the technical terminology here) as a matter of routine for EDA purposes, including those in Cha Sung-Hyuk's incredibly useful taxonomy of distance measures1, as well as some from the Dezas' classic Dictionary of Distances2.

Many of these measures, however, are applied to PDFs, whereas it would be more convenient for me to apply them to CDFs and EDFs. In most cases, I would not be using them to test the fit to the CDF of some arbitrary reference distribution, but as a measure of difference between the known EDFs of two columns. Is it valid to substitute EDFs and CDFs in measures like the Kullback-Leibler and Jensen-Shannon Divergences and if so, does it change their properties in any appreciable way? If not, why so and are there any comparable analogs that we can substitute to measure the same quantities, like relative entropy? I have a passing familiarity with some of these probabilistic and information distances, particularly the KL and Jensen-Shannon, but still have much to learn about them, so this may be a dumb question with an obvious answer; I suspect I may be trying to apply a solution designed for apples to a problem involving oranges, so to speak, but cannot pin it down. Note that this CV question Fitting parametric CDF to ecdf is related since it asks the tangential question on whether there is "some sort of empirical version of Kullback Leibler Divergence," but that side point is never answered.

1 Sung-Hyuk, Cha, 2007, "Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions," pp. 300-307 in International Journal of Mathematical Models and Methods in Applied Sciences, Vol. 1, No. 4. Widely available on the Internet last I knew, including here.

2 Deza, Elena and Deza, Michel Marie, 2006, Dictionary of Distances. Elsevier: Amsterdam.

• You can use 'empirical pdf', which is a discrete distribution over data points with 1/n probability for each. Dec 6, 2017 at 0:27
• What does 'EDF" stand for? Dec 6, 2017 at 0:36
• @BradS. Empirical Distribution Function Dec 6, 2017 at 0:40