# Testing if code is generating random variables consistent with theory

In software development, we often write unit tests to ensure that a particular portion of the code is working as expected. For example, consider the problem of drawing a color at random from Colors=(Red, Green, Blue). Suppose, that we write a function:

drawn_color = draw_color(Colors)


One way to assess whether the code is working correctly or not is to do the following:

tot_iter = 10000
For iter = 1 to tot_iter
drawn_color(iter) = draw_color(Colors)
Next


And then check if the frequencies of the colors in drawn_color are roughly 10000/3. However, the above assessment requires visual inspection and I would rather automate the whole thing so that I can write a test code like so:

If abs(empirical freq - expected freq) < = epsilon then the code is valid.


Two questions:

1. How does epsilon vary as a function of tot_iter?

2. Is there a generic approach to above the issue where the random variables could be drawn from other distributions/densities (e.g., normal, uniform, discrete etc)?

-
You may like to add the goodness-of-fit tag. –  Glen_b Apr 16 '14 at 6:34

First, just to be sure we're clear, a statistical test won't tell you the code is valid. It may sometimes show you when there's a problem. Failure to reject is just saying you couldn't pick it up with that test at that sample size, that time; there's a lot of reasons you might fail to reject.

That said, it does sound like you're seeking a hypothesis test. I'll proceed on that basis, but it may be better to think in terms of decision theory (at the least, an informal consideration of the relative costs of the two types of error may help you choose your sample size and significance level).

If abs(empirical freq - expected freq) < = epsilon then the code is valid.

Presumably, your intent is equivalent to testing the biggest of the three deviations - if you check all three, and conclude you fail if any fail it's equivalent to just checking the worst one.

You could certainly construct a test of this form.

However, a more common test in the case of the multinomial would be the chi-squared goodness of fit test. That is, instead of $\stackrel{_\max}{_i}|O_i-E_i|$, a sum of squared, standardized deviations, $\sum_i (O_i-E_i)^2/E_i$ is often used. If that's just as amenable to you, you may prefer that as something other people are likely to be familiar with (and for which it's relatively easy to find suitable values for $\epsilon$ once you choose your type I error rate). A closely related test (and possibly preferable choice in large sample sizes) is the G-test.

How does epsilon vary as a function of tot_iter?

While the distribution of the mean deviation would probably be reasonably easy to calculate, or at least approximate, I'd probably start by doing a statistic like that via simulation (though in large samples, it should settle down to something you can deal with relatively more simply). That is, simulate the distribution of the proportion when things are as they should be, then specify $\alpha$, the rate at which you're prepared to say "there's a problem" when there is no problem (the Type I error rate, or significance level) and find the cutoff point in the distribution that would cause your rejection rule to reject that proportion of cases.

If you use a more standard test you have a little less work to do.

Is there a generic approach to above the issue where the random variables could be drawn from other distributions/densities (e.g., normal, uniform, discrete etc)?

What you're trying to do is the general area of goodness of fit testing. It's a very large area and there are many, many tests (many dozens to hundreds, really), almost all of which I won't even mention here.

If you have a fully specified continuous distribution (standard normal, standard uniform, Gamma(3,17) etc), you can use a nonparametric test like an Anderson-Darling test or a Kolmogorov-Smirnov test. These are nice in that every continuous distribution can be handled by the same test. [If the distribution is not fully specified, there are various possibilities, but I imagine you're not in that situation.]

If the distribution is discrete, those tests can still work, but you need to adjust the critical values ('the cut off') for the amount of discreteness; I'd probably use simulation. There are other possibilities for testing discrete distributions.

If the distribution is over nominal categories, the aforementioned chi-squared test is common.

Note that getting the univariate distributional shape right is not the only consideration in random number generation. Many tests that consider the multivariate distribution, or serial dependence in some form have been proposed, and entire suites of tests (e.g.1, e.g.2, e.g.3) have been proposed for testing random number generation.

-