I am running some simulations and need a hand with the stats.
Suppose I have $N$ systems. I simulate all of these systems and an event occurs in $x$ of the $N$ systems. I then make a small change to the code such that when I re-run the simulations, the event occurs in $y$ of these systems, where $x <y$.
I want to determine to what confidence I can say that changing the code caused the number of events observed across all systems to increase. The standard in science I've learnt is the 3$\sigma$ confidence - a 99.73% confidence.
So my questions are:
- To what percentage confidence (what sigma level) can I say changing the code increased the number of events?
- How many events $y$ would I need in order for a 3$\sigma$ confidence level?