# How to perform statistical tests on simulated data

I am doing a project trying to investigate the effects of various tactics on the outcome of a naval battle from WW1. I have a simulation which runs many simulations of the battle and records statistics such as hits, ships sunk etc. The data is output in frequency density form with a column for each variable that has been recorded and a row for each number 0-20000. For example if I did 100 simulations and 0 ships were sunk in 12 of them then 0.12 would be in the first row of the ships sunk column. I have 10 or so different samples each representing a different naval tactic and each sample is the result of 100,000 simulations.

My question is how do I conduct statistical tests on data of this form and sample size. I simply need to conduct tests to confirm the significance of differences between two samples so I can draw conclusions from them. Some of the data is not normally distributed also. The simulation samples variables relating to the ships in the battle from a distribution each time it runs so each sample can be considered independent.

Stepping back from your specific query: I question the value of doing hypothesis tests on simulations. In fact, I cannot think of a single case where these p-values would not be useless and misleading, though I'm open to a counterexample. In the question you describe, though, there is clearly no benefit.

Consider that the p-value is in large part a function of sample size. Since you are unlikely to be strongly limited in the number of simulations you can run, finding a significant p-value is simply a matter of devoting sufficient processing power to the question. What would you learn by running 1,000,000 simulations and finding a significant p-value that you did not know with 100,000 simulations and a non-significant one? This objection clearly has relevance to p-value use in other, non-simulated analyses as well, but I think it applies more strongly in the case of simulated data.

A second point against p-value calculations in simulations is made in White et al. (2014): the null hypothesis of no difference between treatments is false by design. The p-value is therefore meaningless to begin with.

Instead, you could simply use the percentages (or something similar) to summarise the difference in effectiveness between tactics in your simulations.

White, J. W., Rassweiler, A., Samhouri, J. F., Stier, A. C., & White, C. (2014). Ecologists should not use statistical significance tests to interpret simulation model results. Oikos, 123(4), 385-388.

• I'm inclined to agree with you. All the tests I have done by hand have given incredibly low p-values just from the sheer size of the sample meaning that I will probably find any difference in mean is significant over the sample. – James Hunt Apr 9 '18 at 17:16
• The paper suggests using ANOVA to compare samples to find effect sizes, something that would be likely useful to me as I am trying to conclude which of the samples represents the best course of action. Is this something that is necessary in my case or can the same be achieved simply by comparing the data, for example by comparing the percentage increase in the mean from the baseline simulation. – James Hunt Apr 9 '18 at 17:48
• @JamesHunt That specifically refers to simulation studies where many factors are being varied; the ANOVA framework is a convenient way to partition the variance explained by the different factors. It's the p-values that are useless. – mkt - Reinstate Monica Apr 9 '18 at 17:51
• @JamesHunt Whether this is useful to you depends on how many factors you are varying in your simulations. – mkt - Reinstate Monica Apr 9 '18 at 17:52