4
$\begingroup$

As with almost all statistical tests in common use, the larger the sample the better the test will be at detecting even the tiniest differences. In my case I want to compare several empirical distributions and see if they are much different. Since the sample sizes of each distribution are quite large, all p-values are very small as expected (very close to zero). Also, in my analysis my aim is that the distributions are (significantly) different, but I struggle finding a way to quantify this other than this test. At the same time, just looking at the p-values doesn't seem very useful for my analysis given the large sample sizes.

Some questions I have regarding this:

  • My plan is to use the KS test statistic as a 'measure' to see how different the distributions are, where a larger value would mean the distributions are more different. Does this interpretation make sense?
  • From what I understand is that the KS test statistic is basically the largest difference between the two empirical cumulative distrubtion functions. Is it hence true that the test statistic ranges from 0 to 1, where 1 would be the biggest evidence for different distributions?
  • Any other suggestions to determine the "degree of difference" between distributions, preferably using a metric/KPI, are welcome.

A related question was asked here, but my question is more focused on the KS test statistic and how I can practically use this (or not) for a "degree of difference" between distributions.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ "As with all statistical tests," - unfortunately, not all tests are necessarily consistent, including even a few in fairly wide use. We might say "almost all tests in common use". $\endgroup$
    – Glen_b
    May 8, 2020 at 14:11
  • $\begingroup$ Updated the question using your feedback. $\endgroup$
    – Amonet
    May 12, 2020 at 12:05
  • 1
    $\begingroup$ @Glen_b-ReinstateMonica - care to give your 2 cents on this question in an answer? I have found your posts very helpful in many threads. $\endgroup$
    – Amonet
    May 16, 2020 at 10:39

3 Answers 3

Reset to default
2
+50
$\begingroup$

About the first two bullets in your question: yes, KS test statistic can be used as a distance metric between different empiric distributions, and yes, it ranges between 0 and 1, where 0 occurs if the two ECDFs are indentical, and 1 if the two samples are completely distinct (the least value of one sample is higher than the maximum value of the other), the more the two samples are "mixed" and then hard to distinguish, the lesser KS test statistic will be.

However, KS test statistic is not a very sensible metric: it only uses the maximum difference between ECDFs, without considering their distance in the rest of their domain, this is the same reason for which KS test is so conservative: ref 1 ref 2. If you are willing to consider other options, there is plenty of choice!

The alternatives

Strictly related metrics to KS test statistic are those of Cramér-von Mises test and Anderson-Darling's. They both consider the whole difference function between the ECDFs. In particular, the latter is generally better regarded.

All these three metrics depend on the order of the observations, and not on their distance, hence, the result is invariant to monotonic transformations of the data.

Another class of distances between probability distributions includes the already cited and very popular KL divergence (or, more properly, Jensen-Shannon divergence), L1 and L2 distance, Hellinger distance. All these metrics require an estimate of the two probability distribution functions, the most simple being histograms, (over the same bins, necessarely). In that case L1 distance is directly related to histogram intersection similarity which has lately known some fortune in data science, for what I've heard. Anyway, JS divergence is the most theoretically founded among all these, from a probability perspective.

All these metrics are computed over the difference between PDFs, so they do not take into consideration distance between data points (of course density estimation can take distance between points into consideration in many ways, like with tails of gaussian kernel for instance).

But my favorite of all between-distributions distances is earth mover distance, which is, in the univariate numeric case, the whole area between two ECDFs. EMD doesn't require estimation of PDFs, and considers not only the order of observations, but also their distance, meaning that it is very sensitive, as long as your observations lies on a sensible metric space. EMD is also very intuitive in its definition, and widely used. Of course, if you don't trust the metric space of your data, you can go for another option.

All these options are "holistic" distance metrics between distributions, in the sense Single Malt meant. Of course you can consider narrower metrics like the simple difference between the arithmetic means of the two samples. That makes a lot of sense for some applications, however, in this answer I have covered more general distances that consider, in one way or another, the whole shape of the two distributions.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for your careful elaboration. One additional question that pops up after reading your answer: do any of these (or another) provide the possibility to focus on only a part of the distribution? For example on the last 10% in the right tail? $\endgroup$
    – Amonet
    May 17, 2020 at 17:45
  • 1
    $\begingroup$ I don't think that any "ready-made" metric focuses on just one tail of the distributions. you can use a metric of your ones, two simple choices being simple distance between 0.9 quantiles of each distribution, and EMD considering only the 10% right tail, which is perfectly feasible (just select 10% higher sample values before computing the distance). $\endgroup$
    – carlo
    May 17, 2020 at 22:44
  • 1
    $\begingroup$ EMD with focus on e.g. 10% of the right tail sounds good, with the advantage of being able to determine the x% myself. Will investigate this further, cheers for your help. $\endgroup$
    – Amonet
    May 18, 2020 at 9:22
1
$\begingroup$

The Kolmogorov–Smirnov test is sensitive to differences in location and shape, and this is one of its strengths, it is in a sense a holistic measure rather than just taking a single metric such as location. But this means that the K-S statistic (D_n) could give high values for differences caused by location or shape and would not distinguish between the two, whereas maybe one is more important to you. Something to counteract multiple comparisons is likely needed, since the more hypotheses that are checked the greater the chance of Type I errors (false positives). The Holm–Bonferroni method, for example, controls the family-wise error rate, and is also fairly simple.

Ranking Kolmogorov–Smirnov statistics may be inadvisable since as a statistic it is itself a random variable, and it may not be clear if a difference (and thus ranking) is real or sampling variation.

For your second question, the answer is yes, the value of zero would occur for two identical empirical distributions, the value of one when the greatest value of one of the distributions is less than the least value of the other.

Would be interesting to get an answer to your third question. Data visualisations may be helpful, although will not provide a single metric.

Improve this answer
$\endgroup$
0
1
$\begingroup$

A better option might be using the Kullback–Leibler divergence, which measure the "distance" between two distribution, p(x) and q(x). When the two distributions are identical, the KL divergence will be 0.

However, you generally can't use KL divergence as a universal distance metric. The most obvious reason is that it is not communicative. That is, D(p(x), q(x)) does not equal D(q(x), p(x)).

It does, however, allow you to compare which of the 2 candidate distributions is closer to the ground truth distribution. That is, if candidate distribution B has KL divergence of 0.1 compared to ground truth distribution A, and candidate distribution C has KL divergence of 0.2 compared to ground truth distribution A, you can conclude that distribution B is closer to A. This is exactly what happens in some optimization problems in ML.

Improve this answer
$\endgroup$
3
  • $\begingroup$ What exactly do you mean by 'you generally can't use KL divergence as a universal distance metric'? Say I have 4 distributions and I want to compare them all (i.e. 6 comparisons), would this be possible using your proposed metric? In this SO thread it is also said it's better not used as a distance metric: stats.stackexchange.com/questions/111445/… $\endgroup$
    – Amonet
    May 12, 2020 at 9:12
  • 1
    $\begingroup$ Suppose you have distribution A,B,C,D, and your task if to find closest or most similar distribution to A, you can then measure the KL divergence D(A,B), D(A,C), D(A,D), and the lowest distance will be the most similar to A. However, if you compute D(B,C), because the reference distribution (B in this case, A in the prior case) is no longer the same, the distance is not meaningful. You can't say A is closer to B than B is closer to C because D(A,B) < D(B,A). In short, it is not to be interpreted as a Euclidean distance, where results are definitive. $\endgroup$
    – Peppershaker
    May 12, 2020 at 18:50
  • 2
    $\begingroup$ Jensen-shannon divergence is more useful tha KL for two equally empiric distributions, but they both require an estimate of pdf, instead of simple ecdf. $\endgroup$
    – carlo
    May 16, 2020 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.