Which is the null hypothesis for testing whether I've broken my simulation?

Question

The situation: I'm writing agent-based computer simulations in which there are random effects which can be biased by various parameters. I run the simulation with the same parameters many times in order to get a distribution of outcomes. All simulations run in the same underlying simulation framework. Now I've changed some source code in the framework. This code randomizes a certain sequence; the first n elements are then chosen from this sequence. This change should not make any meaningful difference in results, I believe, but I'd like to be sure. While I'm not going to test the change for every simulation I've ever run or ever will run, I can at least check the simulations I'm running now to see whether I get different outcome distributions. I ran the same simulation 300 times using the old code and then the new code.

My general question is: Using frequentist statistics, what should my null hypothesis be? The most natural null, I believe, is that the two distributions are the same. However, the dangerous outcome is that I treat the code change as not affecting distributions, when in fact it did affect them. That makes it seem as if the null should be that the distributions are different.

(In order to try to get answers that will be especially helpful, maybe I should say up front that I find some of the traditional ways of talking about choice of null in terms of "what you want to reject" as sometimes unhelpful, counterintuitive, or contrary to what scientists do--as opposed to what they say when they're being fastidious. These answers How to specify the null hypothesis in hypothesis testing fit my understanding pretty well, but I would like to some confirmation about how I'm thinking about my particular problem. In general, my feeling is that the real issue in the choice of null concerns risk. The null should be the alternative which is less costly--for you, for society, whatever--to accept. It's the safe choice. That's why we accept the alternative only if the results are extremely unlikely given the null. In my case, the safe choice is to distrust the intuition that the code change made no difference. Feel free to tell me that I'm wrong, of course! My question is also related to Which one is the null hypothesis?, but I didn't find that discussion sufficiently helpful for my case.)

---

Here are further details which don't matter, I think. You can stop reading unless the following looks relevant:

I did test the "same-distribution" null for several different pairs of outcomes. I used a bootstrapped Kolmogorov-Smirnov tests to get p-values, because the simulations are too complex to know the nature of the underlying distributions, and because the outcomes are roughly discrete (outcomes can vary from -1 to 1, but always converge to near one of several fractional values in practice). In all cases the p-values were greater than .05, so I do not have good evidence the distributions are the same in each case. If I take the null to be that all of these distributions are the same, then I am doing multiple testing--8 tests in all, as it happens, one for each of the numbers generated by a single simulation run--so the relevant level is much lower, .00625: Even better, I think. None of my p-values are anywhere near that cutoff! Do you see what I mean: That this seems like the wrong way to think about testing whether the code change has resulted in a difference in outcome distributions in this case? It's too easy to find that the distributions are the same, and too hard to find that they're different. Should the null be that the distributions are different?

Answer 1

In general, my feeling is that the real issue in the choice of null concerns risk. The null should be the alternative which is less costly--for you, for society, whatever--to accept.

Nulls are not chosen on the basis of 'what's most dangerous' or 'costly'.

You have entirely the wrong idea of the function of a null hypothesis.

http://en.wikipedia.org/wiki/Null_hypothesis

In all cases the p-values were greater than .05, so I do not have good evidence the distributions are the same in each case.

You will never have evidence that 'they are the same'. Only a lack of evidence against it.

If you feel that you are too easily failing to reject the null, consider the possibility that your sample sizes are simply too small. Perhaps a power analysis is in order!

I find some of the traditional ways of talking about choice of null in terms of "what you want to reject"

Well, is it actually statisticians that tend to say that?

I've seen people who aren't statisticians say things like that.

Answer 2

Using a frequentist approach, you can set a null that they are different to a certain degree, and then reject that null. If you do that and reject the null you know they aren't different to that degree. A two tailed test would tell if you they are larger than that degree or less (perhaps to the extent that it is going in another direction), but you can eyeball it to know if that is the case and select to put your degree on the direction of greatest concern (given your post-hoc observation).

K-S tests seem like a good approach in principle. I would recommend that you use the raw K-S statistic from each pair of sims as the input into your analysis and then do something like a one sample t-test of the differences (technically a paired samples t-test, but putting it in the context of a one-sample t makes it more obvious how you can adjust your population mean to reflect the null you actually want to test). In this way you are doing something akin to a random effects meta-analysis of the K-S comparisons and your effects will generalize to the population of your K-S comparisons.

Nulls all too frequently are set to 0. However, it is also reasonable to set them to a prior value that a skeptic might hold. For example here, a skeptic for your sims being the same might hold a null that that they are indeed different (to a given degree), and you want to reject that null and say that they aren't as different as the skeptic believes. Inferential statistics won't ever let you claim they are the same... the best they can do is say that if they are different they are likely only X different (out to some confidence interval). Given all of that... fretting over what the null is in this situation is probably not horribly useful. You probably want to be looking at the confidence interval of the differences between the simulations. The same framework I give in the previous paragraph can be used to this end.

Answer 3

I would recommend to apply the traditional interpretation of the null hypothesis, i.e. that the distributions resulting from both simulation algorithms are identical.

Of course, your concern is another one, i.e that the distributions are different. This is addressed by the power, the complement of the beta error.

It may therefore be advisable to use a two-tailed test, where the sample size is calculated from a-priori power analysis requiring a high power of e.g. 95%. This ensures that the beta error is not more than 5%. Simultaneously the alpha error or p-value should be above the 5% level, but this is a necessary, not a sufficient condition.