So, I wanted to cook up a statistic that was similar to the Wasserstein metric for finite sized samples from distributions on a continuous support that is also invariant to reparameterization of the 1-d support space. I'm familiar with the KS test (and its drawbacks), and aware of the Anderson-Darling and Cramér-von Mises statistics, but this isn't either of those. I'm also aware of several F-divergences, but those only seem to be reparameterization invariant when $Q$ is not en empirical distribution function (I'm happy to be corrected on this point).

First, let $S_1$ be a sample with $M$ continuous random variates, and let $S_2$ be a similar sample with $N$. Label the values from $S_j$ as $x_{ji}$. If we combine the variates into a single set $S$, sort it, we can then construct a curve (a sequence of points connected by straight lines) in the first quadrant unit square as follows (using pseudo-Python):

lastpoint = [0.0, 0.0]
curve = [ lastpoint ]
for x in S:
  newpoint = copy(lastpoint)
  if firstIndex(x) == 1:
    newpoint[0] += 1/N
  else: #firstIndex(x) == 2
    newpoint[1] += 1/M
  lastpoint = newpoint

If $S_i$ was drawn from a distribution with CDF $F_i(x)$, then this curve would, in the infinite sample limit, converge to the parameteric curve $(x,y) = (F_1(t), F_2(t))$ (equivalently, it's the plot of $y=F_2(F_1^{-1}(x))$). Naturally, if $F_1=F_2$ then the curve will be on the line $y=x$, so we can construct statistics from comparing $(F_1(t), F_2(t))$ to $y=x$ and they'll be obviously reparameterization invariant.

The simple statistic I had in mind was total unsigned area enclosed between the curves $y=x$ and $(F_1(t), F_2(t))$ in the unit square. As an integral, that's \begin{align} D &= \int_0^1 \left| x - F_2\left(F_1^{-1}(x)\right)\right| \,\mathrm{d}x. \end{align} This is obviously different from Anderson–Darling and Cramér–von Mises, because it's not quadratic. The construction is so obvious and simple, though, that I can't imagine it hasn't been done already. Sadly, I don't have access to the articles linked in the Anderson-Darling Wikipedia article (EDF Statistics for Goodness of Fit and Some Comparisons seems particularly apt).

My motive? Pure curiosity. Ideally, I'd like it to be useful, but the application I have in mind needs it to be differentiable with respect to some parameters that feed in to generating one of the samples. This statistic defined above would only change when a variate from sample 1 crossed a variate from sample 2 in $x$-space, making it continuous only if either of samples 1 or 2 are infinite in size. It would be possible to make the empirical $F_1$ or $F_2$ continuous by using a smooth empirical distribution function (EDF), instead of a stair-step one, but that requires considering details about the support of the distributions, and that's outside of the scope of this question.


1 Answer 1


What you describe is reminiscent of "Probability Plotting". See D’Agostino & Stephens (1986), section 2.3. In that section (2.3.1), they define a probability plot as that of $G^{-1}\left(F(x_i)\right)$ against $x_i$ for ordered $X$. Their initial recommendation (2.3.3) is to æsthetically determine how "close" it is to a straight line. However, for certain distributions, the slope and intercept have direct relationships with the distributional parameters. While they do not give your statistic a name, in Chapter 5, they describe regression methods for finding parameters which are intended to minimize the distance between the probability plot and $y=x$ (see 5.1). To be fair, regression is minimizing squared error and not absolute error.

  • 1
    $\begingroup$ Interesting. This one works in the dual space (i.e. it's not probability versus probability, but $x_1$ vs $x_2$), which would make it maximally reparameterization covariant. I'm also not seeing how this version could be used to compare two ecdfs (maybe using a similar procedure of merge, sort, step through combined list updating $x$ and $y$ as you go). It would probably be more sensitive to variations on the tails, though, which can be good and bad... $\endgroup$
    – Sean Lake
    Commented Nov 9, 2022 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.