1
$\begingroup$

What is the practical difference between Wasserstein metric and Jensen-Shannon divergence? Wasserstein metric is also referred to as Earth mover's distance.

From Wikipedia:

Wasserstein metric is a distance function defined between probability distributions on a given metric space M. Intuitively, if each distribution is viewed as a unit amount of earth (soil) piled on M, the metric is the minimum "cost" of turning one pile into the other, which is assumed to be the amount of earth that needs to be moved times the mean distance it has to be moved.

and

Jensen-Shannon divergence is a method of measuring the similarity between two probability distributions. It is based on the Kullback–Leibler divergence, with some notable (and useful) differences, including that it is symmetric and it always has a finite value. The square root of the Jensen–Shannon divergence is a metric often referred to as Jensen-Shannon distance.

I've seen JSD used in machine learning, but not so much Wasserstein metric even though it improves GAN. Is there a good guideline on when to use one or the other?

$\endgroup$
3
  • 1
    $\begingroup$ regarding Wasserstein and KL, I'm fond of my answer here: stats.stackexchange.com/a/295729/150025 which is a partial duplicate-or at least has some overlap-with this one. $\endgroup$ – Lucas Roberts Oct 2 '20 at 0:49
  • $\begingroup$ thanks for pointing out there that the Wasserstein metric does not require both measures to be on the same probability space, whereas KL divergence requires both measures to be defined on the same probability space. does this also hold for Jensen-Shannon divergence (i've edited the question to it instead)? $\endgroup$ – develarist Oct 2 '20 at 1:25
  • $\begingroup$ I think that JS does not require explicitly. If I understand the M = (P+Q)/2 notation from wikipedia article correctly then M is a smoothing so that the support of M is the union of the supports of P, Q. However, that might lead to some 0 probabilities and I'm not clear how the JS variant handles those. I'm less familiar with JS overall than KL and Wasserstein. $\endgroup$ – Lucas Roberts Oct 3 '20 at 17:47
1
$\begingroup$

Following examples by Arjovsky et al (2017) and Kolouri et al (2018), Kolouri et al (2019) shows a simple example in the supplementary material comparing the Jensen-Shannon divergence with the Wasserstein distance.

enter image description here

As can be seen the JS divergence fails to provide a useful gradient when the distributions are supported on non-overlapping domains.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.