# Is there any probability distance that preserves all properties of a metric?

In studying Kullback–Leibler distance, there are two things we learn very quickly is that it does not respect neither the triangle inequality nor the symmetry, required properties of a metric.

My question is whether there is any metric of probability density functions that fulfil all the constraints of a metric.

-
To focus on probability densities is to focus on the "wrong" object. As for metrics, there are the "classical" ones, e.g., Lévy (and the related Ky Fan metric on random variables), Wasserstein along with ones closer in spirit to KL, e.g., Jensen-Shannon divergence. Though mostly overlooked historically, note that in the original KL paper, the KL divergence was indeed symmetric (though still not a metric). –  cardinal Feb 17 '14 at 17:54
@cardinal, well, I'm not so much in the field, can you please suggest the "right" object? –  J. C. Leitão Feb 17 '14 at 17:57
J.C.: Sorry, the comment box became too small for all I was trying to fit in there. I should have elaborated. The cumulative distribution function turns out to be a more general and natural object of study. :-) –  cardinal Feb 18 '14 at 0:44
@cardinal why? ;) –  J. C. Leitão Feb 18 '14 at 8:49

Take a look at this paper that covers a wide range of popular metrics on the space of probability measure. My personal favorites are the total variation distance and $L^2$ Wasserstein distance (earth mover distance).

-
That is a good paper - especially figure 1. I'm saving a copy of that for future reference. –  Pat Feb 17 '14 at 17:57

I believe that the Earth Mover's Distance, also known as the Wasserstein metric, is an example which meets your requirements.

-

There are some modifications to the KL divergence that make it acquire some of the metric properties (though not all).

For example, the Jeffrey’s divergence modifies the KL divergence to make it symmetric.

There are some special cases see [1]: "Unfortunately, traditional measures based on the Kullback–Leibler (KL) divergence and the Bhattacharyya distance do not satisfy all metric axioms necessary for many algorithms. In this paper we propose a modification for the KL divergence and the Bhattacharyya distance, for multivariate Gaussian densities, that transforms the two measures into distance metrics."

[1] K. Abou-Moustafa and F. Ferrie, "A Note on Metric Properties for Some Divergence Measures: The Gaussian Case," JMLR: Workshop and Conference Proceedings 25:1–15, 2012.

-