# How to measure the distance (or divergence - not sure) between data and a probability distribution?

If I have generated a set of random data and I wish to measure how well these data fit, e.g. a uniform probability distribution, what are the standard ways to do that?

I am not very experienced with this sort of data processing. One thing that comes to mind is Kullback-Leibler divergence, but there you measure distance between two p.d.f.s. Here I have one p.d.f. and then a set of data I would like to compare it to, but which I can probably at best represent as a histogram. I have seen people mentioning the Kolmogorov-Smirnov statistic. What that be suitable here?

There are goodness of fit tests (many, many), including the Kolmogorov-Smirnov test you mentioned; many goodness of fit test statistics can be seen as a measure of discrepancy between data and some distribution.

The most obvious ones to consider first are the empirical-CDF-based tests, of which the Kolmogorov-Smirnov test is the simplest.

The Kolmogorov-Smirnov test statistic is the largest distance between the hypothesized cdf and the ECDF of the data (or sometimes a standardized version of it)

There are many others based on the ECDF. One example is the Cramer-von Mises test which (at least for the uniform) corresponds to a sum of squares of vertical distances between the two cdfs.

Another related measure is the Anderson-Darling statistic, which weights by the inverse of the variance of the ECDF.

There are many other kinds of tests. One common one is the the Shapiro-Wilk test, for example. The Shapiro-Wilk (and related tests) can also be treated as a measure of discrepancy between the distribution of the data and some hypothesized distribution (in the case of the Shapiro-Wilk test, that's the normal distribution). The Shapiro-Wilk, however, is invariant to changes in mean and variance.

A full coverage of the goodness of fit territory could take a book -- indeed has taken several.

• It might be very nice also to say about what is standardized version of the ECDF, since it is mentioned. Mar 4, 2014 at 11:05