I have a multivariate distribution (for which I know the parameters) that I simulate data from. I then fit several distributions to this simulated data using several different approaches (similar to MLE) and I want to compare how well my fitted distributions compare to the true distribution.

Most of the methods I have found so far regarding "comparing distributions" seem to really be comparing samples from two different processes. I'm curious if there is any research that provides a metric of how close two known distributions are to one another. My initial thought is to consider something like an integral of the absolute difference between the two distributions over the entire domain, but that seems computationally difficult to compute and maybe not the best approach.

  • $\begingroup$ In the univariate case, that is exactly how you do it, and it is called Kolmogorov-Smirnov test. That statistic, namely the maximum absolute distance between the distributions, has a known distribution that can be used in a test. In the multivariate test, though, I've no idea! $\endgroup$ – Felipe Gerard Feb 26 '16 at 13:38
  • $\begingroup$ en.wikipedia.org/wiki/Kolmogorov–Smirnov_test - the last paragraph $\endgroup$ – German Demidov Feb 26 '16 at 13:40

You might find the notions of Hellinger and Kullback-Leibler divergence interesting: https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence https://en.wikipedia.org/wiki/Hellinger_distance They can be applied to multivariate distributions, although actual computations can get difficult in some cases.

  • $\begingroup$ '@Jacek Podlewski' are you aware of any implemenations of KL or Hellinger in higher dimentions? The latter is based on Radon-Nikodym derivatives, is this easy to code? $\endgroup$ – mjs Jan 7 '20 at 14:39

Peacock test is what you are looking for. It's an extension of popular Kolmogorov-Smirnov test to n >= 2 dimensions. See for example the paper by Press/Teukolsky about it wth C implementation details or Lopes et al. for a general overview on that topic.

[Edit] Just found a great paper comparing number of metrics/distances by Gibbs & Su ON CHOOSING AND BOUNDING PROBABILITY METRICS

  • 1
    $\begingroup$ The OP specifically says he/she is NOT trying to compare samples from two different distribution, and specifically asks for a measure of how different two known multivariate distributions are. $\endgroup$ – logistic Dec 16 '19 at 14:53
  • $\begingroup$ @logistic yes, you are right of course. I will remove my answer but first wanted to clarify that e.g. other KS related answers are wrong as well, correct? Measures like the KL divergence or Helliger distance in contrast work with probability distributions whose PDF are known. Any comments would be very appreciated. $\endgroup$ – mjs Dec 17 '19 at 8:32
  • $\begingroup$ @logistic On the other hand, all the other answers I could find do talk about differences between distributions in KS context, stackoverflow.com/questions/52471839/… or Wikipedia en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test $\endgroup$ – mjs Dec 17 '19 at 8:50
  • $\begingroup$ My mind went to using something like KL divergence to measure difference between distributions, but I don't know enough to actually submit an answer. $\endgroup$ – logistic Dec 17 '19 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.