I develop GNU Parallel. For each release I would like to test whether the version I am about to release has a memory leak.
I can measure the memory usage. Unfortunately the usage varies slightly given the same input. The memory usage is: base + leak. The base varies slightly and there is an expected leak of 1 bit per job. What I want to detect is if there is a bigger leak than the expected leak.
For one release the base is between, say, 13050 and 13150 kB and for another release it can be between 15000 and 15500 kB. They are more or less normally distributed.
It is very easy to see if there is a memory leak: Just compare runs of 1000 jobs to 1000000 jobs and if there is a leak of 1 byte/job then the usage will increase by 1000000 bytes which is bigger than the range of the base memory usage.
But running 1000000 jobs will take around 10000 seconds (1 job takes around 0.01 sec to run). So is there a faster way in which I can determine if there is likely a bigger than expected leak?
I am thinking of running 10 runs with 1000 jobs and 10 runs with 2000 jobs and comparing the distributions of memory usage of these. If the distributions are significantly different then there is a leak. How do I do that?
Intuitively I can see that you need fewer runs and fewer jobs if the leak is big (say, 500 bytes per job) than if the leak is small (say, 2 bytes per job), and thus it should be possible to stop much earlier. How do I figure out how many runs and how many jobs I should run to get a 99% certainty?
(I am a novice at statistics, R and Python, and advanced in Perl. So a solution will have to show real code, not just refer to some statistical methods; also code that gives a "yes/no" is preferred over "eyeballing it").
((Ideally I want 2 pieces of code: One that I can tell "You have 3 minutes to run. After that I want to know how certain you are about a memleak of 1 byte, 10 bytes, 100 bytes, and 1000 bytes", and the second version that show me the certainties continously while it is running more runs to gather more datapoints.))