2
$\begingroup$

I have an interesting application I’d like some advice on.

A common task when working with stochastic systems in an engineering context is testing for regressions and improvements in performance. First, it’s important to have a confidence one is collecting enough data points to have an accurate assessment of the distribution of performance. And second, it’s important to have a good strategy for comparing those distributions to identify any discernible performance changes. However, I’m not sure there’s a widely accepted solution for how to tackle these challenges. Or if there is, I don’t know about it yet.

I’ll describe the problem I’m facing in more detail. I have a process whose objective is to reach a certain goal in as little time as possible. The process is stochastic, so the runtime has some random variations. In fact, the variance of the runtime could be fairly large. Every day I collect one “sample” by running the process to completion for $N$ repetitions and recording the elapsed time per repetition, thus collecting $N$ data points per sample. My recording precision is 0.1 seconds. After I’ve collected a sample I’d like to compare it with yesterday’s sample to measure if there’s been a change in performance.

What do I mean when I say "a change in performance?" I'm interested in significant changes to the average runtime (the mean) and to the variation in the runtimes (variance). Part of the issue is I'm not quite sure how to define "significance" in this context. My definition for now is any change in the mean greater than four times the recording precision, and any change in the variance greater than a few percent.

There's an additional complication: I'd like to find a method where I don't need to make assumptions about special features of the distribution, like the specific scale or shape. The reason for this is I'm performing the process I described above for several stochastic processes which each have distinct properties. For example, one produces a bimodal distribution; another produces a one-tailed distribution. I had tried setting up problem-specific thresholds for the mean and variance, but tuning them proved quite difficult, especially since each has high variance.

Given the problem I’ve just described, I have two questions:

1. How do I know when I’ve collected enough data points per sample to make an accurate comparison of two sample distributions? I would love to collect millions of datapoints per sample but it takes several hours to collect 1000 datapoints, so I’m dealing with a classic tradeoff.

2. What are some traditionally used methods for comparing two samples in order to detect any significant change in the mean and variance of the stochastic process? This is assuming I’ve collected enough data points to have fairly accurate views of the distributions of the two samples.

I would include the work I’ve done so far to tackle this problem, but I don’t want this post to be any longer. My knowledge of statistics is fairly basic, which is why I’m appealing to the good folks at crossvalidated for advice or pointers to related materials. I've found a few related posts but nothing which described my problem exactly. Thank you for reading and I appreciate anything you can give me.

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ The required $n$ depends on the distribution of the data (e.g. when just comparing means, more variable means you need more data) and the precise comparison being made and on your criteria for 'accurate'... so that's rather a broad question. Consideration of the different ways you might compare could fill books on its own. Your second question has similar difficulties, with words like 'best' which depends on what your criteria for 'best' are. $\endgroup$
    – Glen_b
    Feb 27 '14 at 20:30
  • $\begingroup$ What information would you need for the word 'best' to no longer be ambiguous? $\endgroup$
    – sharshofski
    Feb 27 '14 at 20:35
  • $\begingroup$ At the least, a clear indication of what you want to optimize. $\endgroup$
    – Glen_b
    Feb 27 '14 at 20:37
  • $\begingroup$ Ok. I would like to find a method of comparing samples such that I'm able to detect significant changes in the mean and variance. I'm aware there are many ways to approach this problem, but I'm not aware of all of the ones which are most likely to be useful. I'd be interested in discussing any proposals which seem likely to produce good results, which is why I wrote this post. Is that any more clear? $\endgroup$
    – sharshofski
    Feb 27 '14 at 20:47
  • $\begingroup$ Also, @ your first point: I am aware more data is required for a distribution with high variance. I want to know if there's a clean way to have high confidence my sample is a good representation of the actual distribution, or if not to therefore know I should increase n. $\endgroup$
    – sharshofski
    Feb 27 '14 at 20:47
0
$\begingroup$

You should look into the literature of control charts and quality control. If changes are occurring abruptly, look into methods for detecting changepoints.

For means (and variances), you could plot running estimates based on some window of observations, maybe with exponentially decreasing weights. Also much useful information in the many comments.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.