13
$\begingroup$

In studying Kullback–Leibler distance, there are two things we learn very quickly is that it does not respect neither the triangle inequality nor the symmetry, required properties of a metric.

My question is whether there is any metric of probability density functions that fulfil all the constraints of a metric.

Improve this question
$\endgroup$
4
  • $\begingroup$ To focus on probability densities is to focus on the "wrong" object. As for metrics, there are the "classical" ones, e.g., Lévy (and the related Ky Fan metric on random variables), Wasserstein along with ones closer in spirit to KL, e.g., Jensen-Shannon divergence. Though mostly overlooked historically, note that in the original KL paper, the KL divergence was indeed symmetric (though still not a metric). $\endgroup$
    – cardinal
    Commented Feb 17, 2014 at 17:54
  • 1
    $\begingroup$ @cardinal, well, I'm not so much in the field, can you please suggest the "right" object? $\endgroup$
    – Jorge Leitao
    Commented Feb 17, 2014 at 17:57
  • 3
    $\begingroup$ J.C.: Sorry, the comment box became too small for all I was trying to fit in there. I should have elaborated. The cumulative distribution function turns out to be a more general and natural object of study. :-) $\endgroup$
    – cardinal
    Commented Feb 18, 2014 at 0:44
  • $\begingroup$ @cardinal why? ;) $\endgroup$
    – Jorge Leitao
    Commented Feb 18, 2014 at 8:49

4 Answers 4

Reset to default
20
$\begingroup$

Take a look at this paper that covers a wide range of popular metrics on the space of probability measure. My personal favorites are the total variation distance and $L^2$ Wasserstein distance (earth mover distance).

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ That is a good paper - especially figure 1. I'm saving a copy of that for future reference. $\endgroup$
    – Pat
    Commented Feb 17, 2014 at 17:57
  • $\begingroup$ @Ramanujan updated to an arxiv link. $\endgroup$
    – John Jiang
    Commented Dec 8, 2021 at 22:11
7
$\begingroup$

I believe that the Earth Mover's Distance, also known as the Wasserstein metric, is an example which meets your requirements.

Improve this answer
$\endgroup$
1
$\begingroup$

There are some modifications to the KL divergence that make it acquire some of the metric properties (though not all).

For example, the Jeffrey’s divergence modifies the KL divergence to make it symmetric.

There are some special cases see [1]: "Unfortunately, traditional measures based on the Kullback–Leibler (KL) divergence and the Bhattacharyya distance do not satisfy all metric axioms necessary for many algorithms. In this paper we propose a modification for the KL divergence and the Bhattacharyya distance, for multivariate Gaussian densities, that transforms the two measures into distance metrics."

[1] K. Abou-Moustafa and F. Ferrie, "A Note on Metric Properties for Some Divergence Measures: The Gaussian Case," JMLR: Workshop and Conference Proceedings 25:1–15, 2012.

Improve this answer
$\endgroup$
0
$\begingroup$

I think that answer to the question is possible. Because, recently in 2017 R. Farhadian showed that there is a probability on a heuristic sub set of integers that it is a metric. for his work, see the following link: http://journals.univ-danubius.ro/index.php/oeconomica/article/view/4010

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.