Testing implementation of Anderson-Darling test for uniform RV

Question

I am trying to write unit tests for a whole mess of statistics code. Some of the unit tests take the form: generate a sample following a null hypothesis, use code to get a p-value under that null, repeat hundreds of times, then look at all the p-values: if they are reasonably uniform, then the code passes. I usually check if the ratio of p-values < $\alpha$ is near $\alpha$ for $\alpha = 0.01, 0.05, 0.1$. But I am usually also interested in whether the p-values output deviate from uniformity. I usually test this with the Anderson-Darling test.

Here is where I have a circularity problem: how can I unit test my Anderson-Darling code? I can easily feed it uniformly generated variables, and get a p-value, repeat hundreds of times, but then I just have a bunch of p-values. I can q-q plot them, but I'm more interested in an automatic unit test I can run. What are some basic sanity checks I can implement automatically? there is the naive check of ratio of p-values < $\alpha$ noted above. I can also implement a Kolmogorov-Smirnov test. What else can I easily check for?

(I realize this question may seem hopelessly pedantic or naive or subject to infinite regress...)

edit some additional ideas:

1. test the code on $\frac{i}{n}$ for $i = 1,2,...,n$, for different values of $n$. Presumably I can compute, by hand, the p-value for the A-D test in this case.
2. compute $n$ p-values by feeding many uniform samples to the code $n$ times, then regress the order statistics of the p-values, $p_{(i)}$ vs $i/n$, to get $p_{(i)} = \beta_1 i/n + \beta_0$ and test the null $\beta_1 = 1, \beta_0 = 0$. Presumably I can simplify this test by hand in such a way that inspection reveals it to be correct.
3. make sure the code is invariant with respect to permutation of the input. (duh)

Answer 1

You could test your Anderson-Darling code using data that is generated from an external library. However, you then run into the issue of how to test/trust the external library. At some point you have to trust that well established libraries are error free and that their output can be relied on.

Once you have the Anderson-Dalring code tested against data generated from an external library the circularity will be broken and you can rely on your own code if it passes the Anderson-Darling tests. The same will hold for the K-S (I presume Kolmogorov–Smirnov) test.

Answer 2

I can q-q plot them, but I'm more interested in an automatic unit test I can run.

If a visual inspection of a q-q plot would suffice, then you could calculate an entropy measure such as the Gini coefficient and accept the test after allowing for some tolerance for deviation from the 45 degree line. For more on comparing distributions in general, you could look to the reldist package in R for ideas.

Though not directly applicable, I think you'll also find the method presented by Cook, Gelman & Rubin interesting:

• Validation of Software for Bayesian Models Using Posterior Quantiles