I am trying to write unit tests for a whole mess of statistics code. Some of the unit tests take the form: generate a sample following a null hypothesis, use code to get a p-value under that null, repeat hundreds of times, then look at all the p-values: if they are reasonably uniform, then the code passes. I usually check if the ratio of p-values < $\alpha$ is near $\alpha$ for $\alpha = 0.01, 0.05, 0.1$. But I am usually also interested in whether the p-values output deviate from uniformity. I usually test this with the Anderson-Darling test.

Here is where I have a circularity problem: how can I unit test my Anderson-Darling code? I can easily feed it uniformly generated variables, and get a p-value, repeat hundreds of times, but then I just have a bunch of p-values. I can q-q plot them, but I'm more interested in an automatic unit test I can run. What are some basic sanity checks I can implement automatically? there is the naive check of ratio of p-values < $\alpha$ noted above. I can also implement a Kolmogorov-Smirnov test. What else can I easily check for?

(I realize this question may seem hopelessly pedantic or naive or subject to infinite regress...)

edit some additional ideas:

  1. test the code on $\frac{i}{n}$ for $i = 1,2,...,n$, for different values of $n$. Presumably I can compute, by hand, the p-value for the A-D test in this case.
  2. compute $n$ p-values by feeding many uniform samples to the code $n$ times, then regress the order statistics of the p-values, $p_{(i)}$ vs $i/n$, to get $p_{(i)} = \beta_1 i/n + \beta_0$ and test the null $\beta_1 = 1, \beta_0 = 0$. Presumably I can simplify this test by hand in such a way that inspection reveals it to be correct.
  3. make sure the code is invariant with respect to permutation of the input. (duh)
  • $\begingroup$ Could you give a couple of examples of what your code does? $\endgroup$ – csgillespie Aug 10 '10 at 9:42
  • $\begingroup$ briefly pv = AD_test_uni(xs) takes a sample vector xs, which are restricted to the range $[0,1]$ and returns a p-value, pv under the null that xs are drawn from the Uniform distribution. I can then use this elsewhere to test other tests which spit out a p-value. $\endgroup$ – shabbychef Aug 10 '10 at 16:28

You could test your Anderson-Darling code using data that is generated from an external library. However, you then run into the issue of how to test/trust the external library. At some point you have to trust that well established libraries are error free and that their output can be relied on.

Once you have the Anderson-Dalring code tested against data generated from an external library the circularity will be broken and you can rely on your own code if it passes the Anderson-Darling tests. The same will hold for the K-S (I presume Kolmogorov–Smirnov) test.

  • $\begingroup$ There is indeed an infinite regress problem here. However, it is certainly not the case that all external libraries are error-free (I have found such errors in the past). Because all of my tests ultimately rest on this A-D test, I thought it might be worthy of extra scrutiny. $\endgroup$ – shabbychef Aug 10 '10 at 16:44
  • $\begingroup$ Perhaps, you can use multiple external libraries whose code base does depend on each other. It is unlikely that independent libraries have the same errors which should eliminate some of your concerns. $\endgroup$ – user28 Aug 10 '10 at 16:49

I can q-q plot them, but I'm more interested in an automatic unit test I can run.

If a visual inspection of a q-q plot would suffice, then you could calculate an entropy measure such as the Gini coefficient and accept the test after allowing for some tolerance for deviation from the 45 degree line. For more on comparing distributions in general, you could look to the reldist package in R for ideas.

Though not directly applicable, I think you'll also find the method presented by Cook, Gelman & Rubin interesting:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.