2
$\begingroup$

I'm trying to choose a district metric that falls between 0 to 1 and lets me compare the distance between a uniform probability distribution and any given probability distribution (could be random, could be uniform, could have all probability mass at one value, etc.).

How I've been computing distance currently is by basically computing the average L1-norm distance between each bin in the two distributions. However, I haven't been able to find any published metric that does this. It seems like similar to the total variation distance and the Hellinger distance, but neither of those average across the total number of bins. Averaging avoids weaknesses of total variation that occur when, for example, given a true distribution of Unif[0,1] total variation distance = 1 if my experimental distribution is Unif[0.1,1] or Unif[0.9,1] (discretized into 10 bins). If I average, I get a distance near 0 for Unif[0,1] and closer to 1 for Unif[0.9,1].

Jenson-Shannon divergence or chi-squared seem like they could work, but I'm not sure how to choose.

The Wasserstein metric could work, but since it's not between 0 to 1, it'll make comparing distributions across different distances difficult (e.g., for a different scenario my true distribution might be Unif[0,5] with 50 instead of 10 bins).

I would appreciate if someone could 1) point me to a metric that's the average L1 distance between histogram indices (or tell me why this is a poor choice), and 2) Help me figure out how to choose between the above metrics (or point me to intuitive discussions of how to choose between them -- the mostly measure theory based papers I've found comparing the metrics unfortunately don't give me intuition on which to choose for my problem).

Improve this question
$\endgroup$
2
  • $\begingroup$ What do you mean by a "district metric"? $\endgroup$
    – whuber ♦
    Commented Oct 15, 2017 at 14:37
  • $\begingroup$ I think "district metric" is a typo and they meant "distance metric". $\endgroup$
    – dennisobrien
    Commented Aug 8, 2018 at 21:00

1 Answer 1

Reset to default
1
$\begingroup$

When you look for a metric for the distance between 2 probability measures don't you need to make sure they are measures over the same sample space and over the same sigma-algebra? You are comparing UNIF[0,1] and UNIF[.1,1]. These are not comparable since the sample spaces are [0,1] and [.1,1] respectively. That means there are many sets for which you cannot calculate the probability for both. For example the probability of [.01, .09] can only be calculated for UNIF[0,1]. That interval is not in the domain of the probability measure for UNIF[.1,1].

Improve this answer
$\endgroup$
4
  • $\begingroup$ There are many distance measures that do not impose such a requirement. For instance, the $L^p$ distance between the two distribution functions is always available, regardless of the essential support of the measures. By definition, the probability of any interval can be computed for any distribution. Under a Uniform$(0.1,1)$ distribution, for instance, the probability of $[0.01,0.09]$ is zero. $\endgroup$
    – whuber ♦
    Commented Nov 6, 2018 at 17:40
  • $\begingroup$ Isn't a probability measure only defined for events in the sample space? In particular, it is only defined for elements of the sigma-algebra. The interval [.01,.09] is not an event if our sample space is [.1,1]. If we extended the sample space to [0,1] and defined the probability of [.01,.09] = 0 then P{[.01,.09]} would be well defined. $\endgroup$
    – Gene
    Commented Nov 6, 2018 at 20:21
  • $\begingroup$ If we instead consider the true sample space to be (-inf, +inf) and impose two probability measures with different supports then I agree that any two probability measures on (-inf, +inf) with the same sigma algebra are comparable. In that case P{[.01,.09]} = 0 under the measure UNIF[.1,1]. $\endgroup$
    – Gene
    Commented Nov 6, 2018 at 20:34
  • $\begingroup$ We're discussing distributions in this thread, not random variables. $\endgroup$
    – whuber ♦
    Commented Nov 6, 2018 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.