How to perform statistical tests on simulated data

Question

I am doing a project trying to investigate the effects of various tactics on the outcome of a naval battle from WW1. I have a simulation which runs many simulations of the battle and records statistics such as hits, ships sunk etc. The data is output in frequency density form with a column for each variable that has been recorded and a row for each number 0-20000. For example if I did 100 simulations and 0 ships were sunk in 12 of them then 0.12 would be in the first row of the ships sunk column. I have 10 or so different samples each representing a different naval tactic and each sample is the result of 100,000 simulations.

My question is how do I conduct statistical tests on data of this form and sample size. I simply need to conduct tests to confirm the significance of differences between two samples so I can draw conclusions from them. Some of the data is not normally distributed also. The simulation samples variables relating to the ships in the battle from a distribution each time it runs so each sample can be considered independent.

Answer 1

Stepping back from your specific query: I question the value of doing hypothesis tests on simulations. In fact, I cannot think of a single case where these p-values would not be useless and misleading, though I'm open to a counterexample. In the question you describe, though, there is clearly no benefit.

Consider that the p-value is in large part a function of sample size. Since you are unlikely to be strongly limited in the number of simulations you can run, finding a significant p-value is simply a matter of devoting sufficient processing power to the question. What would you learn by running 1,000,000 simulations and finding a significant p-value that you did not know with 100,000 simulations and a non-significant one? This objection clearly has relevance to p-value use in other, non-simulated analyses as well, but I think it applies more strongly in the case of simulated data.

A second point against p-value calculations in simulations is made in White et al. (2014): the null hypothesis of no difference between treatments is false by design. The p-value is therefore meaningless to begin with.

Instead, you could simply use the percentages (or something similar) to summarise the difference in effectiveness between tactics in your simulations.

White, J. W., Rassweiler, A., Samhouri, J. F., Stier, A. C., & White, C. (2014). Ecologists should not use statistical significance tests to interpret simulation model results. Oikos, 123(4), 385-388.