5
$\begingroup$

I am trying to understand what exactly the distance between two distributions using Wasserstein distance means.

I have two samples coming from two distribution: a ground truth one and its empirical realization. I know that the Wasserstein distance can be used to quantify the difference between the two distributions. My question is when do we consider the distance between these distributions "small" enough? or what does this number mean ? say we obtain 0.25 for the distance. What does that tell us ?

I think the answer of this question comes down to understand what does the distance exactly quantify (and this question goes beyond the simple interpretation of the definition :the minimum cost if we want to obtain the first distribution by transporting the probability mass in second one )

I am including a python example here and I appreciate an answer with concrete examples

from scipy.stats import wasserstein_distance
wasserstein_distance([0, 1, 3], [5, 6, 8])

(note : the scipy implementation works only on 1d PDs)

$\endgroup$
3
  • 1
    $\begingroup$ The Wikipedia page en.wikipedia.org/wiki/Wasserstein_metric is helpful in this regard; basically, the metric quantifies how much mass must be moved around, and how far, to turn one distribution into the other. $\endgroup$
    – jbowman
    Mar 24, 2020 at 15:59
  • $\begingroup$ Thank you for your answer, I understand the conceptual meaning of what this metric quantifies (as I mentioned in the post) but I would like to understand what the numbers really mean and how would I know that my distributions are close ? (for instance does 1 mean that the the pds close ? or 0.1 ? or maybe 0.001 ? and does the size of the two samples matter in this case ? for instance if we have two samples coming from two pds and each one of them has size 10000 and the distance between them is 1, does that mean they are close ? versus two samples with size 100 and have the same distance? ) $\endgroup$
    – Steve
    Mar 24, 2020 at 16:08
  • $\begingroup$ Think of the real line. Are 1 and 5 close? $\endgroup$
    – Tobsn
    Nov 4, 2020 at 21:23

2 Answers 2

5
$\begingroup$

Wasserstein (or EMD), once you multiply it by your bandwith, measures the "work" necessary to transform one distribution into another (by solving the optimal transport problem). Roughly that is the integral difference between the two distributions, multiplied by the distance between their centers (NOTE: this is an approximation only for the purpose of giving a simple explanation here, but Wassertein makes NO USE of centers/average of the distributions and IT DOES USE a distance matrix that is user-provided and can be asymmetric or use non-linear steps -- The figure attached makes use of a symmetric distance matrix built with linear steps equal to the bin size of the distributions).

The wikipedia page explains everything with adequate math definitions: en.wikipedia.org/wiki/Wasserstein_metric

Below you can see the metrics with respect the reference BOLD BLUE.

enter image description here

$\endgroup$
5
  • 1
    $\begingroup$ where in the Wasserstein formula does it measure the "distance between their centers"? I thought the distance matrix makes no reference to the centers $\endgroup$
    – develarist
    Dec 3, 2020 at 11:08
  • $\begingroup$ It does NOT, you are perfectly right. I edited my answer, while still trying to just provide a simple explanation. $\endgroup$ Dec 3, 2020 at 11:46
  • $\begingroup$ how can the wasserstein measure be modified, so that it identifies right-wards distances to the right skew, and distances moved to the left skew of a distribution. because without modifying it, it will treat both directional movements as if they were the same not realizing when is in the negative direction, and the other in the positive direction $\endgroup$
    – develarist
    Dec 3, 2020 at 14:33
  • $\begingroup$ I don't know if it's possible, I never needed/tried this. It is possible though, using an assymetric distance matrix, to get the correct distance in periodic conditions: for example, using the attached plot, consider the system is now periodic between x = [0, 10]. Then you can get the correct distance of 3 between pink and brown by modifying the EMD underlying dist matrix. See also: stackoverflow.com/questions/56939150/… $\endgroup$ Dec 3, 2020 at 17:28
  • $\begingroup$ thanks for editing your answer above to say that the wasserstein distance doesn't impute centers, but it still says "the distance between their centers" just before your edit $\endgroup$
    – develarist
    Dec 3, 2020 at 17:40
5
$\begingroup$

As others have mentioned, the Wasserstein metric measures how much work is required to transform one distribution to another.

However, I think the following is a more inspirational view of this metric. By definition, the Wasserstein metric operates on two distributions over the same metric space. The Wasserstein metric "lifts" the metric on the underlying metric space to a metric on distributions on that metric space. Thus, the distances produced by the Wasserstein metric are intimately influenced by the metric you're used to in the underlying metric space.

One method of computing the Wasserstein distance between distributions $\mu, \nu$ over some metric space $(X, d)$ is to minimize, over all distributions $\pi$ over $X\times X$ with marginals $\mu,\nu$, the expected distance $d(x, y)$ where $(x, y)\sim\pi$. Here you can clearly see how this metric is simply an expected distance in the underlying metric space.

Moreover, it may help to know that the Wasserstein distance is merely a special case of the more general optimal transport cost. Optimal transport theory actually allows you to define these distances with respect to an arbitrary cost function rather than the distance, and even distances between distributions over completely different metric spaces. But the really nice thing about them in my opinion is how the metric over distributions is so nicely tied to functions on the underlying spaces.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.