I have an interesting application I’d like some advice on.
A common task when working with stochastic systems in an engineering context is testing for regressions and improvements in performance. First, it’s important to have a confidence one is collecting enough data points to have an accurate assessment of the distribution of performance. And second, it’s important to have a good strategy for comparing those distributions to identify any discernible performance changes. However, I’m not sure there’s a widely accepted solution for how to tackle these challenges. Or if there is, I don’t know about it yet.
I’ll describe the problem I’m facing in more detail. I have a process whose objective is to reach a certain goal in as little time as possible. The process is stochastic, so the runtime has some random variations. In fact, the variance of the runtime could be fairly large. Every day I collect one “sample” by running the process to completion for $N$ repetitions and recording the elapsed time per repetition, thus collecting $N$ data points per sample. My recording precision is 0.1 seconds. After I’ve collected a sample I’d like to compare it with yesterday’s sample to measure if there’s been a change in performance.
What do I mean when I say "a change in performance?" I'm interested in significant changes to the average runtime (the mean) and to the variation in the runtimes (variance). Part of the issue is I'm not quite sure how to define "significance" in this context. My definition for now is any change in the mean greater than four times the recording precision, and any change in the variance greater than a few percent.
There's an additional complication: I'd like to find a method where I don't need to make assumptions about special features of the distribution, like the specific scale or shape. The reason for this is I'm performing the process I described above for several stochastic processes which each have distinct properties. For example, one produces a bimodal distribution; another produces a one-tailed distribution. I had tried setting up problem-specific thresholds for the mean and variance, but tuning them proved quite difficult, especially since each has high variance.
Given the problem I’ve just described, I have two questions:
1. How do I know when I’ve collected enough data points per sample to make an accurate comparison of two sample distributions? I would love to collect millions of datapoints per sample but it takes several hours to collect 1000 datapoints, so I’m dealing with a classic tradeoff.
2. What are some traditionally used methods for comparing two samples in order to detect any significant change in the mean and variance of the stochastic process? This is assuming I’ve collected enough data points to have fairly accurate views of the distributions of the two samples.
I would include the work I’ve done so far to tackle this problem, but I don’t want this post to be any longer. My knowledge of statistics is fairly basic, which is why I’m appealing to the good folks at crossvalidated for advice or pointers to related materials. I've found a few related posts but nothing which described my problem exactly. Thank you for reading and I appreciate anything you can give me.