# Testing implementation of Anderson-Darling test for uniform RV

I am trying to write unit tests for a whole mess of statistics code. Some of the unit tests take the form: generate a sample following a null hypothesis, use code to get a p-value under that null, repeat hundreds of times, then look at all the p-values: if they are reasonably uniform, then the code passes. I usually check if the ratio of p-values < $\alpha$ is near $\alpha$ for $\alpha = 0.01, 0.05, 0.1$. But I am usually also interested in whether the p-values output deviate from uniformity. I usually test this with the Anderson-Darling test.

Here is where I have a circularity problem: how can I unit test my Anderson-Darling code? I can easily feed it uniformly generated variables, and get a p-value, repeat hundreds of times, but then I just have a bunch of p-values. I can q-q plot them, but I'm more interested in an automatic unit test I can run. What are some basic sanity checks I can implement automatically? there is the naive check of ratio of p-values < $\alpha$ noted above. I can also implement a Kolmogorov-Smirnov test. What else can I easily check for?

(I realize this question may seem hopelessly pedantic or naive or subject to infinite regress...)

1. test the code on $\frac{i}{n}$ for $i = 1,2,...,n$, for different values of $n$. Presumably I can compute, by hand, the p-value for the A-D test in this case.
2. compute $n$ p-values by feeding many uniform samples to the code $n$ times, then regress the order statistics of the p-values, $p_{(i)}$ vs $i/n$, to get $p_{(i)} = \beta_1 i/n + \beta_0$ and test the null $\beta_1 = 1, \beta_0 = 0$. Presumably I can simplify this test by hand in such a way that inspection reveals it to be correct.
3. make sure the code is invariant with respect to permutation of the input. (duh)
• Could you give a couple of examples of what your code does? – csgillespie Aug 10 '10 at 9:42
• briefly pv = AD_test_uni(xs) takes a sample vector xs, which are restricted to the range $[0,1]$ and returns a p-value, pv under the null that xs are drawn from the Uniform distribution. I can then use this elsewhere to test other tests which spit out a p-value. – shabbychef Aug 10 '10 at 16:28