The situation: I'm writing agent-based computer simulations in which there are random effects which can be biased by various parameters. I run the simulation with the same parameters many times in order to get a distribution of outcomes. All simulations run in the same underlying simulation framework. Now I've changed some source code in the framework. This code randomizes a certain sequence; the first n elements are then chosen from this sequence. This change should not make any meaningful difference in results, I believe, but I'd like to be sure. While I'm not going to test the change for every simulation I've ever run or ever will run, I can at least check the simulations I'm running now to see whether I get different outcome distributions. I ran the same simulation 300 times using the old code and then the new code.
My general question is: Using frequentist statistics, what should my null hypothesis be? The most natural null, I believe, is that the two distributions are the same. However, the dangerous outcome is that I treat the code change as not affecting distributions, when in fact it did affect them. That makes it seem as if the null should be that the distributions are different.
(In order to try to get answers that will be especially helpful, maybe I should say up front that I find some of the traditional ways of talking about choice of null in terms of "what you want to reject" as sometimes unhelpful, counterintuitive, or contrary to what scientists do--as opposed to what they say when they're being fastidious. These answers How to specify the null hypothesis in hypothesis testing fit my understanding pretty well, but I would like to some confirmation about how I'm thinking about my particular problem. In general, my feeling is that the real issue in the choice of null concerns risk. The null should be the alternative which is less costly--for you, for society, whatever--to accept. It's the safe choice. That's why we accept the alternative only if the results are extremely unlikely given the null. In my case, the safe choice is to distrust the intuition that the code change made no difference. Feel free to tell me that I'm wrong, of course! My question is also related to Which one is the null hypothesis?, but I didn't find that discussion sufficiently helpful for my case.)
Here are further details which don't matter, I think. You can stop reading unless the following looks relevant:
I did test the "same-distribution" null for several different pairs of outcomes. I used a bootstrapped Kolmogorov-Smirnov tests to get p-values, because the simulations are too complex to know the nature of the underlying distributions, and because the outcomes are roughly discrete (outcomes can vary from -1 to 1, but always converge to near one of several fractional values in practice). In all cases the p-values were greater than .05, so I do not have good evidence the distributions are the same in each case. If I take the null to be that all of these distributions are the same, then I am doing multiple testing--8 tests in all, as it happens, one for each of the numbers generated by a single simulation run--so the relevant level is much lower, .00625: Even better, I think. None of my p-values are anywhere near that cutoff! Do you see what I mean: That this seems like the wrong way to think about testing whether the code change has resulted in a difference in outcome distributions in this case? It's too easy to find that the distributions are the same, and too hard to find that they're different. Should the null be that the distributions are different?