# How to measure the shift between two cumulative distribution functions (CDFs)?

How to measure the shift between two cumulative distribution functions (CDFs)? Specifically, in the image below, how meaningful is the shaded area? It is supposed to measure the shift between the green and the red CDF. On the x-axis, the value of the variable is given in arbitrary units, while on the y-axis, it's the cumulative density. I'm concerned that, given it's all cumulative, the differences in the lower left keep contributing to the area further up, so that there is a much stronger weight on the differences between values that are leftmost on the x-axis, where cumulation begins. Am I right? Am I wrong? If so, where is my misunderstanding? • The shaded (signed) area is the difference in expectations of the two distributions. There are no "weights" in any parts of the area--it's just an area. "Signed" means that areas where the red curve is to the left of the green curve are subtracted from the total. (No such areas appear in this image).
– whuber
Aug 23 '17 at 12:42
• Orthogonal comments, picky if you like, but in graphics details can matter a lot. 1. Difficulty distinguishing red and green is a common problem many people have; here it doesn't matter because the two curves would be distinct whatever the colours, but even for this kind of problem, red and green curves that crossed could be a puzzle. Red and blue or orange and blue are much less problematic. 2. Diagonal shading is meaningless here. Appropriate shading should just be monochrome. Aug 23 '17 at 13:17
• @Nick: You're correct, I'll use red/blue in the end. Aug 25 '17 at 3:59

Think about what a CDF represents in terms of probability. Let the variables on the x-axis be referred to as $x$ and y-axis values be referred to as $y$. By definition the cumulative distribution function is showing the probability that a variable is less than or equal to $x$. More specifically, if you look at $x=0$ for each curve the CDF is telling you: $P_{\text{red}}(X \leq 0) \approx 0.5$ and $P_{\text{green}}(X \leq 0) \approx 0.7$.

Your question is a little vague so I will answer it in two parts.

How meaningful is the difference at a particular point?: Let's assume $X$ represents the difference in points from an average test score (with a negative value representing below average and positive value representing above average). Let the green curve represent boys and red curve represent girls. Now, $P_{\text{red}}(X \leq 0) \approx 0.5$ and $P_{\text{green}}(X \leq 0) \approx 0.7$ tells us that the probability of a boy scoring below average is higher than the probability of a girl scoring below average. If we look at the CDF as whole (green always above red) this suggests in your sample population, girls score higher than boys. Whether or not this result is statistically significant is yet to be determined.

How meaningful is the difference overall? (edited as a response to @whuber) : This depends on how you use it. For instance, if the green CDF represented the CDF of some reference distribution and the red CDF was an empirical sample distribution, then the point by point vertical differences can be used in a Kolmogorov–Smirnov test for equality between the two distributions.

The fact that the green "leads" the red and the two curves are similarly shaped contribute to the fact that the green is always above the red, but this does not necessarily have to be the case. Consider that your populations do not come from the same underlying distribution. In this case the shape of the CDF would differ and the fact that the green "leads" the red would not necessarily result in the green always being above the red. For example, here are various CDFs of a logistic distribution (from Wikipedia) Notice that the red curve in the plot above "leads" (starts from a nonzero value) before the rest of the curves do, but ultimately end up below most of the curves as the x-values approach x=20.

• The K-S test does not use areas: it uses maximum vertical discrepancies between a CDF and an empirical CDF.
– whuber
Aug 23 '17 at 13:03
• @whuber agreed. I suppose I should be more clear in my response. Aug 23 '17 at 13:07
• See my comments on colours, specifically red and green, on the original question. Aug 23 '17 at 14:25
• joceratops: Many thanks as well! The underlying data are actually more complex. In your analogy of boys and girls and their scores, you can think of the boys and girls as actually being of different age. Calculating the shift of "all boys" versus "all girls", as is currently done, ignores this information completely; is there a way of taking it into account? Aug 24 '17 at 5:56
• In fact, a better analogy to the data I have is children of different age (that's the x-axis, older to younger from left to right)), and their height. Children older than 10 years, say, are green; the others are red. The shift confirms that older children are taller. Nevertheless, the data I have may be misleading: There may e.g. exist a weird, opposing trend within the teenagers, among which the rule could be "the younger the taller". Aug 24 '17 at 5:57

The absolute value of this area is $$\int_{x=-\infty}^\infty \lvert F(x) - G(x)\rvert \,\mathrm{d}x,$$ which note – at least for continuous distributions – is exactly equal to $$\int_{x=-\infty}^\infty \lvert F^{-1}(x) - G^{-1}(x)\rvert \,\mathrm{d}x.$$ In one dimension, the latter is the 1-Wasserstein distance, the 1-Kantorovich distance, or the "earth-mover's distance." It is quite a reasonable distance between probability distributions, which is easy to compute between one-dimensional distributions based on their cdfs.

For multivariate distributions, there is a natural extension (not based on CDFs, which become hard to work with), most commonly defined based on optimal transport. You can think of it this way: think of each density function as a pile of dirt. The amount of dirt that you need to move to transform one density into another is exactly this distance. This leads to the name of "earth-mover's distance."

It's not entirely obvious at first that this distance corresponds to the difference in area between CDFs. But imagine doing this for two point mass distributions, one at $$x$$ and one at $$x'$$; the area between their CDFs is a rectangle, with area $$1 \times \lvert x - x' \rvert$$, exactly the amount that you need to move the "dirt." You can then envision doing the same thing for a collection of point masses, getting a series of rectangles that you add up. When you go to continuous distributions in the limit, you get the integral written above, and it should hopefully make sense that these are the same thing.

The traditional way to estimate this distance from samples is by directly computing this transportation problem with a linear program, though there are more recent fast approximations.

The beautiful Kantorovich-Rubinstein duality also applies to this distance. This relation has this year has led to an explosion of interest in the Wasserstein distance among the deep learning community, via this paper which uses it for generative modeling. The distance has also been popular in computer vision applications for decades.

• +1. For clarity, please note that your expression is not an "integral of this area": it simply is the (absolute) area.
– whuber
Aug 23 '17 at 16:14
• Dougal: Many thanks for your answer! Thinking in the framework of "moving dirt" is making things much clearer and intuitive. However,for this description to work, you need to visualize the data as histograms (piles of dirt), right? Or, is there also an intuitive analogy for the shaded area in my figure, with direct reference to the CDFs? (See also my followup below to joceratops). Aug 24 '17 at 6:00
• I added a description of how to visualize this as a CDF area. Hopefully it makes sense; in f not, I can try to come back and make some pictures later. :) Aug 24 '17 at 8:35
• @Dougal: Thanks. But how to explain this to a biologist, in terms s/he understands (that's the task I'd like to accomplish, being not so familiar with the topic myself, at least not any more). Moving dirt I can explain ;-) Aug 24 '17 at 9:08
• @Dougal Yes, I see this now. Intuitively, the first expression (absolute difference of CDFs) integrates "horizontally" by adding up vertical slices between $F(x)$ and $G(x)$, while the second expression integrates "vertically" by adding up horizontal slices between $F^{-1}(x)$ and $G^{-1}(x)$. May 14 '20 at 19:20