I am building software components that are going to be used by multiple users simultaneously. At the current point in time, these components are known bad. That is, if I simulate multiple users simultaneously interacting with each component long enough I know the component will fail. I cannot determine that a failure happened until the end of the simulation.
The result of the simulation gives me two things:
- A success or failure indicator, and
- The actual number of times an error occurred (although I don't know when in the simulation the error actually happened)
I also have no idea what kind of distribution the errors have because changing the application to record the failures at the time that they happened would likely change the distribution itself.
My goal is to determine the smallest sample size I can use to achieve a target confidence level CL at a target confidence interval CI so that I can automatically have these components tested the necessary number of iterations. I'll then fix the components but leave the automated component tests to give me some confidence that the components aren't broken in the future.
How can I determine the smallest number of iterations (samples) the simulator needs to run for in order to reproduce a failure at confidence level CL% given a confidence interval of CI%? Or, if I can't equate the number of iterations to samples, what approach might I take to achieve the above goal?
If it helps, I can run the simulator any number of times at any number of iterations in order to have a sufficient sample size.
As I do not know enough statistics to know exactly what questions to ask or what research to do, let me explain my current plan:
- Run the simulator for X iterations
- If the simulator does not fail, repeat step 1 with a larger X until the simulation fails.
- (As the simulation failed at X iterations) Run the simulator 1000 times with X iterations to determine what I suspect would be something akin to the confidence level. (i.e., if it failed 950 of the 1000 times I would assume a confidence level of 95%.)
- If the confidence level is not sufficiently high, repeat starting at step 1 one with a larger X (assuming I'll get a better confidence level).
I doubt the above is mathematically sound and would like to fix that.
It feels like the unknown distribution makes it impossible to determine the likelihood of a failure per some number of iterations. It also feels like I could use the actual number of times an error occurred within a simulation to my advantage - perhaps by calculating variance over repeated simulations.
# iterations # failed simulations / 1000 simulations 2^n (0 <= n <= 6) 0 128 662 256 837 512 916 1024 974 2048 734 4096 779 8192 891 16384 953 32768 968 65536 991 131072 999 2^n (n >= 18) 1000
As you can see, at 2048 iterations the simulation failed significantly fewer times than at 1024 iterations, which to me suggests some large variance.
I'm fairly capable with general mathematics, but quite uninformed on the statistics front, so gearing answers in that direction would help.