Testing statistical software What techniques/approaches are useful in testing statistical software? I'm particularly interested in programs that do parametric estimation using maximum likelihood.
Comparing results to those from other programs or published sources is not always possible since most of the time when I write a program of my own it is because the computation I need is not already implemented in an existing system.
I am not insisting on methods which can guarantee correctness. I would be happy with techniques that can catch some fraction of errors.
 A: One useful technique is monte carlo testing.  If there are two algorithms that do the same thing, implement both, feed them random data, and check that (to within a small tolerance for numerical fuzz) they produce the same answer.  I've done this several times before:


*

*I wrote an efficient but hard to implement $O(N\ log\ N)$ implementation of Kendall's Tau B.  To test it I wrote a dead-simple 50-line implementation that ran in $O(N^2)$.  

*I wrote some code to do ridge regression.  The best algorithm for doing this depends on whether you're in the $n > p$ or $p > n$ case, so I needed two algorithms anyhow.  
In both of these cases I was implementing relatively well-known techniques in the D programming language (for which no implementation existed), so I also checked a few results against R.  Nonetheless, the monte carlo testing caught bugs I never would have caught otherwise.
Another good test is asserts.  You may not know exactly what the correct results of your computation should be, but that doesn't mean that you can't perform sanity checks at various stages of the computation.  In practice if you have a lot of these in your code and they all pass, then the code is usually right.
Edit:  A third method is to feed the algorithm data (synthetic or real) where you know at least approximately what the right answer is, even if you don't know exactly, and see by inspection if the answer is reasonable.  For example, you may not know exactly what the estimates of your parameters are, but you may know which ones are supposed to be "big" and which ones are supposed to be "small".
A: Bruce McCullough had a bit of a cottage industry in assessing statistical software (in the widest sense; he also tested Microsoft Excel. And found it wanting). Two papers that illustrate part of his approach are here and here.
A: A lot of detail is given by the President of StataCorp, William Gould, in this Stata Journal article.1 It is a very interesting article about quality control of statistical software.
