2
$\begingroup$

What is the practical difference between Wasserstein metric and Jensen-Shannon divergence? Wasserstein metric is also referred to as Earth mover's distance.

From Wikipedia:

Wasserstein metric is a distance function defined between probability distributions on a given metric space M. Intuitively, if each distribution is viewed as a unit amount of earth (soil) piled on M, the metric is the minimum "cost" of turning one pile into the other, which is assumed to be the amount of earth that needs to be moved times the mean distance it has to be moved.

and

Jensen-Shannon divergence is a method of measuring the similarity between two probability distributions. It is based on the Kullback–Leibler divergence, with some notable (and useful) differences, including that it is symmetric and it always has a finite value. The square root of the Jensen–Shannon divergence is a metric often referred to as Jensen-Shannon distance.

I've seen JSD used in machine learning, but not so much Wasserstein metric even though it improves GAN. Is there a good guideline on when to use one or the other?

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ regarding Wasserstein and KL, I'm fond of my answer here: stats.stackexchange.com/a/295729/150025 which is a partial duplicate-or at least has some overlap-with this one. $\endgroup$
    – Lucas Roberts
    Commented Oct 2, 2020 at 0:49
  • $\begingroup$ thanks for pointing out there that the Wasserstein metric does not require both measures to be on the same probability space, whereas KL divergence requires both measures to be defined on the same probability space. does this also hold for Jensen-Shannon divergence (i've edited the question to it instead)? $\endgroup$
    – develarist
    Commented Oct 2, 2020 at 1:25
  • $\begingroup$ I think that JS does not require explicitly. If I understand the M = (P+Q)/2 notation from wikipedia article correctly then M is a smoothing so that the support of M is the union of the supports of P, Q. However, that might lead to some 0 probabilities and I'm not clear how the JS variant handles those. I'm less familiar with JS overall than KL and Wasserstein. $\endgroup$
    – Lucas Roberts
    Commented Oct 3, 2020 at 17:47

1 Answer 1

Reset to default
5
$\begingroup$

Following examples by Arjovsky et al (2017) and Kolouri et al (2018), Kolouri et al (2019) shows a simple example in the supplementary material comparing the Jensen-Shannon divergence with the Wasserstein distance.

enter image description here

As can be seen the JS divergence fails to provide a useful gradient when the distributions are supported on non-overlapping domains.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Finally i found intuitive explanation, thank you $\endgroup$
    – Yaro Nosa
    Commented Jul 21, 2023 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.