I am running some simulations and need a hand with the stats.

Suppose I have $N$ systems. I simulate all of these systems and an event occurs in $x$ of the $N$ systems. I then make a small change to the code such that when I re-run the simulations, the event occurs in $y$ of these systems, where $x <y$.

I want to determine to what confidence I can say that changing the code caused the number of events observed across all systems to increase. The standard in science I've learnt is the 3$\sigma$ confidence - a 99.73% confidence.

So my questions are:

  1. To what percentage confidence (what sigma level) can I say changing the code increased the number of events?
  2. How many events $y$ would I need in order for a 3$\sigma$ confidence level?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.