1
$\begingroup$

I am running some simulations and need a hand with the stats.

Suppose I have $N$ systems. I simulate all of these systems and an event occurs in $x$ of the $N$ systems. I then make a small change to the code such that when I re-run the simulations, the event occurs in $y$ of these systems, where $x <y$.

I want to determine to what confidence I can say that changing the code caused the number of events observed across all systems to increase. The standard in science I've learnt is the 3$\sigma$ confidence - a 99.73% confidence.

So my questions are:

  1. To what percentage confidence (what sigma level) can I say changing the code increased the number of events?
  2. How many events $y$ would I need in order for a 3$\sigma$ confidence level?
Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ See significance of difference between two counts. $\endgroup$
    – Scortchi ♦
    Commented Aug 25, 2015 at 14:07
  • $\begingroup$ Thanks for the link but I'm afraid I really don't understand that very well at all $\endgroup$
    – Tom
    Commented Aug 25, 2015 at 16:07
  • 1
    $\begingroup$ Conditioning on the total no. counts reduces the problem to inference about binomial proportions: see Confidence interval for Bernoulli sampling. But on re-reading your question I wonder whether it really does concern Poisson counts. If $x$ is the total count of an event that can either occur (once) or not in each of the $N$ systems then you're comparing two binomial distributions: see onlinecourses.science.psu.edu/stat414/node/268. $\endgroup$
    – Scortchi ♦
    Commented Aug 26, 2015 at 9:55

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.