13
$\begingroup$

I would like to know what the most powerful way of comparing two (or more) discrete distributions is.

I know that the Kolmogorov-Smirnov test could be used (if corrected for the discrete ecdfs), and/or a chi-squared test, and other summary statistics could be compared (mean/variance/skewness &c), but is there a more powerful test along the lines of the Cramér–von_Mises test? There is unlikely to be much deviation across the whole of the discrete distributions, so I'd like the test to have as much power as possible, a situation that C-vM would be best suited for if the distributions were samples from a continuous distribution.

Some background:

Multiple machines generate strings of a fixed length with an added 'tail' (of an random integer number - say between 0 and 250 - of special characters). Changing the environment the machines sit in may (or may not) change the tail-length distributions.

Taking all the strings from the machines at different time points will give a time-varying distribution of special character tail lengths.

I'd like to know if we can test whether there are significant changes in the tail-length distribution over the time course.

Improve this question
$\endgroup$

3 Answers 3

Reset to default
9
$\begingroup$

It looks like you have a clear understanding of all the available tests. What I would suggest is if you would get the book, "Goodness-of-Fit-Techniques" by Ralph B. D'Agostino. It provides the background on the tests as well as good examples. Here a link to a pdf http://www.gbv.de/dms/ilmenau/toc/04207259X.PDF.

One thing you could add to list of methods to compare probability densities is Kullback–Leibler divergence. If you are a scientist you will see the relation between this method and entropy but that's not required. Here is just the introduction (explanation) of the approach which I took from Wikipedia "In probability theory and information theory, the Kullback–Leibler divergence (also information divergence, information gain, relative entropy, KLIC, or KL divergence) is a measure of the difference between two probability distributions P and Q..." This is covered in Matlab.

An "off-shoot" of Kullback-Leibler is the Jensen–Shannon divergence for probability distributions (this is a more common approach to comparing probability distrubtions (PD). This looks at the similarity of the PDs. Should be numerous references on this and is also covered in MatLab.

These methods aren't covered in the reference I gave you above.

I also found a good paper for you to look at. https://www.math.hmc.edu/~su/papers.dir/metrics.pdf, "On Choosing and Bounding Probability Metrics." The metrics covered are: Discrepancy, Hellinger distance, Kullback-Leibler divergence, Kolmogorov metric, Levy metric, Prokhorov metric, Separation distance, Total variation distance, Wasserstein (or Kantorovich) metric, and chi squared metric.

I should also mentioned the "q-q plot" (where q refers to quantile) as a simple way to compare 2 probability distributions (or compare data to a probability distribution).

Also one test that was left out earlier is the Anderson-Darling test. Here are two references for it: (1) http://www.win.tue.nl/~rmcastro/AppStat2013/files/lectures23.pdf (2) https://asaip.psu.edu/Articles/beware-the-kolmogorov-smirnov-test. The second reference goes over the problems you could encounter with Kolmogorov-Smirnov test.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Multiple references are appreciated. $\endgroup$
    – deps_stats
    Oct 18, 2021 at 23:29
4
$\begingroup$

You could use a Cramer-von Mises statistic (or anything similar to it that you think makes sense) with a discrete distribution as long as you treat it as a resampling test rather than relying on the tables that assume a continuous distribution.

Improve this answer
$\endgroup$
2
  • $\begingroup$ can you go into more detail as to how the resampling test would be conducted? $\endgroup$
    – Vass
    Sep 14, 2018 at 21:56
  • 2
    $\begingroup$ Either as a permutation test, or possibly a bootstrap. For example with a permutation test, under the null the populations are identical so the group labels would be arbitrary; the distribution of your test statistic would not be changed by swapping (permuting) group labels across observations. You simply count the proportion of those (across all possible permutations) that are as or more extreme (more consistent with the alternative) than your observed sample. $\endgroup$
    – Glen_b
    Sep 14, 2018 at 23:52
3
$\begingroup$

You can have a look on optimal transport between distributions. It is often used in Computer Vision (under the name Earth Mover Distance) to compare histograms (seminal paper). It is also used in Machine Learning, cf. the work from Cuturi (e.g. this paper) and from Marti et al..

Improve this answer
$\endgroup$
1
  • $\begingroup$ in simple terms, how would optimal transport between distribution work in this case? $\endgroup$
    – Vass
    Sep 14, 2018 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.