Testing that a sample corresponds to an arbitrary distribution

Question

I'm writing a set of unit test for a function that draws random numbers from an arbitrary distribution, defined as a PDF (example of such a function can be seen here). I am confused about the proper way of obtaining a good confidence that the function output indeed follows that specified distribution.

I know that proving equality is much harder that disproving it, but good approximation will do.

An answer on this site suggests using Pearson's chi-squared test for similar purposes in the case of discrete variable. Here, I'm dealing with continuous variable, for which the PDF is defined using paired vectors $X$ and $P$ of final sizes.

Answer 1

There are many goodness-of-fit tests, as they are known, available for continuous distributions. If your hypothesized distribution happens to be Normal, you can use the Shapiro-Wilk test (shapiro.test in R.) Other distribution-specific tests exist as well. More general-purpose tests include the Anderson-Darling, Cramer-Von Mises, and Kolmogorov-Smirnov tests. Of these three, the Anderson-Darling is the most sensitive to misfit in the tails.

Many other goodness-of-fit tests exist; the choice between them depends at least in part upon what "features" of misfitting are most important for you to detect. Of course, if you have a random number generator and can generate hundreds of thousands of numbers, any halfway decent test will detect all but the slightest deviations from the hypothesized distribution. If you are attempting to detect programming errors, this would seem to me to be good enough, since, a priori, we'd expect a typical programming error to induce a rather large deviation from the desired outputs. On the other hand, if you mistype a nine-digit hardcoded number in the 9th digit, you might well induce a tiny deviation from the desired outputs that is virtually indetectable even with a huge sample size.

Also note that there are many ways for a random number generator to fail to generate random numbers (or pseudo-random numbers if you prefer that terminology) other than having the distribution of the outputs fail to match the desired distribution. One example is serial correlation of the outputs. Higher order lack of independence is also possible; the infamous randu generator is a cautionary example to all random number generator researchers.