Minimum sample size to achieve a specific confidence level and interval

Question

I am building software components that are going to be used by multiple users simultaneously. At the current point in time, these components are known bad. That is, if I simulate multiple users simultaneously interacting with each component long enough I know the component will fail. I cannot determine that a failure happened until the end of the simulation.

The result of the simulation gives me two things:

1. A success or failure indicator, and
2. The actual number of times an error occurred (although I don't know when in the simulation the error actually happened)

I also have no idea what kind of distribution the errors have because changing the application to record the failures at the time that they happened would likely change the distribution itself.

My goal is to determine the smallest sample size I can use to achieve a target confidence level CL at a target confidence interval CI so that I can automatically have these components tested the necessary number of iterations. I'll then fix the components but leave the automated component tests to give me some confidence that the components aren't broken in the future.

How can I determine the smallest number of iterations (samples) the simulator needs to run for in order to reproduce a failure at confidence level CL% given a confidence interval of CI%? Or, if I can't equate the number of iterations to samples, what approach might I take to achieve the above goal?

If it helps, I can run the simulator any number of times at any number of iterations in order to have a sufficient sample size.

As I do not know enough statistics to know exactly what questions to ask or what research to do, let me explain my current plan:

1. Run the simulator for X iterations
2. If the simulator does not fail, repeat step 1 with a larger X until the simulation fails.
3. (As the simulation failed at X iterations) Run the simulator 1000 times with X iterations to determine what I suspect would be something akin to the confidence level. (i.e., if it failed 950 of the 1000 times I would assume a confidence level of 95%.)
4. If the confidence level is not sufficiently high, repeat starting at step 1 one with a larger X (assuming I'll get a better confidence level).

I doubt the above is mathematically sound and would like to fix that.

It feels like the unknown distribution makes it impossible to determine the likelihood of a failure per some number of iterations. It also feels like I could use the actual number of times an error occurred within a simulation to my advantage - perhaps by calculating variance over repeated simulations.

Data:

# iterations      # failed simulations / 1000 simulations
2^n (0 <= n <= 6) 0
128               662
256               837
512               916
1024              974
2048              734
4096              779
8192              891
16384             953
32768             968
65536             991
131072            999
2^n (n >= 18)     1000

As you can see, at 2048 iterations the simulation failed significantly fewer times than at 1024 iterations, which to me suggests some large variance.

I'm fairly capable with general mathematics, but quite uninformed on the statistics front, so gearing answers in that direction would help.

Answer 1

this may or may not be helpful for you, but Diaz-Emperanza (1996, 2000) has published on how many replications a Monte Carlo study has so that a certain width of a confidence interval for a parameter is achieved.

Here are the two papers, which can both be accessed at http://ideas.repec.org/e/pda47.html

Ignacio Díaz-Emparanza, 2000. "Is a small Monte Carlo analysis a good analysis? Checking the size, power and consistency of a simulation-based test," Econometrics 0004005, EconWPA.

Ignacio Dmaz-Emparanza, 1996. "Selecting the Number of Replications in a Simulation Study," Econometrics 9612006, EconWPA.