Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

In studying Kullback–Leibler distance, there are two things we learn very quickly is that it does not respect neither the triangle inequality nor the symmetry, required properties of a metric.

My question is whether there is any metric of probability density functions that fulfil all the constraints of a metric.

share|improve this question
    
To focus on probability densities is to focus on the "wrong" object. As for metrics, there are the "classical" ones, e.g., Lévy (and the related Ky Fan metric on random variables), Wasserstein along with ones closer in spirit to KL, e.g., Jensen-Shannon divergence. Though mostly overlooked historically, note that in the original KL paper, the KL divergence was indeed symmetric (though still not a metric). –  cardinal Feb 17 at 17:54
1  
@cardinal, well, I'm not so much in the field, can you please suggest the "right" object? –  J. C. Leitão Feb 17 at 17:57
1  
J.C.: Sorry, the comment box became too small for all I was trying to fit in there. I should have elaborated. The cumulative distribution function turns out to be a more general and natural object of study. :-) –  cardinal Feb 18 at 0:44
    
@cardinal why? ;) –  J. C. Leitão Feb 18 at 8:49

3 Answers 3

up vote 13 down vote accepted

Take a look at this paper that covers a wide range of popular metrics on the space of probability measure. My personal favorites are the total variation distance and $L^2$ Wasserstein distance (earth mover distance).

share|improve this answer
2  
That is a good paper - especially figure 1. I'm saving a copy of that for future reference. –  Pat Feb 17 at 17:57

I believe that the Earth Mover's Distance, also known as the Wasserstein metric, is an example which meets your requirements.

share|improve this answer

There are some modifications to the KL divergence that make it acquire some of the metric properties (though not all).

For example, the Jeffrey’s divergence modifies the KL divergence to make it symmetric.

There are some special cases see [1]: "Unfortunately, traditional measures based on the Kullback–Leibler (KL) divergence and the Bhattacharyya distance do not satisfy all metric axioms necessary for many algorithms. In this paper we propose a modification for the KL divergence and the Bhattacharyya distance, for multivariate Gaussian densities, that transforms the two measures into distance metrics."

[1] K. Abou-Moustafa and F. Ferrie, "A Note on Metric Properties for Some Divergence Measures: The Gaussian Case," JMLR: Workshop and Conference Proceedings 25:1–15, 2012.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.