2
$\begingroup$

I have an interesting application I’d like some advice on.

A common task when working with stochastic systems in an engineering context is testing for regressions and improvements in performance. First, it’s important to have a confidence one is collecting enough data points to have an accurate assessment of the distribution of performance. And second, it’s important to have a good strategy for comparing those distributions to identify any discernible performance changes. However, I’m not sure there’s a widely accepted solution for how to tackle these challenges. Or if there is, I don’t know about it yet.

I’ll describe the problem I’m facing in more detail. I have a process whose objective is to reach a certain goal in as little time as possible. The process is stochastic, so the runtime has some random variations. In fact, the variance of the runtime could be fairly large. Every day I collect one “sample” by running the process to completion for $N$ repetitions and recording the elapsed time per repetition, thus collecting $N$ data points per sample. My recording precision is 0.1 seconds. After I’ve collected a sample I’d like to compare it with yesterday’s sample to measure if there’s been a change in performance.

What do I mean when I say "a change in performance?" I'm interested in significant changes to the average runtime (the mean) and to the variation in the runtimes (variance). Part of the issue is I'm not quite sure how to define "significance" in this context. My definition for now is any change in the mean greater than four times the recording precision, and any change in the variance greater than a few percent.

There's an additional complication: I'd like to find a method where I don't need to make assumptions about special features of the distribution, like the specific scale or shape. The reason for this is I'm performing the process I described above for several stochastic processes which each have distinct properties. For example, one produces a bimodal distribution; another produces a one-tailed distribution. I had tried setting up problem-specific thresholds for the mean and variance, but tuning them proved quite difficult, especially since each has high variance.

Given the problem I’ve just described, I have two questions:

1. How do I know when I’ve collected enough data points per sample to make an accurate comparison of two sample distributions? I would love to collect millions of datapoints per sample but it takes several hours to collect 1000 datapoints, so I’m dealing with a classic tradeoff.

2. What are some traditionally used methods for comparing two samples in order to detect any significant change in the mean and variance of the stochastic process? This is assuming I’ve collected enough data points to have fairly accurate views of the distributions of the two samples.

I would include the work I’ve done so far to tackle this problem, but I don’t want this post to be any longer. My knowledge of statistics is fairly basic, which is why I’m appealing to the good folks at crossvalidated for advice or pointers to related materials. I've found a few related posts but nothing which described my problem exactly. Thank you for reading and I appreciate anything you can give me.

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ The required $n$ depends on the distribution of the data (e.g. when just comparing means, more variable means you need more data) and the precise comparison being made and on your criteria for 'accurate'... so that's rather a broad question. Consideration of the different ways you might compare could fill books on its own. Your second question has similar difficulties, with words like 'best' which depends on what your criteria for 'best' are. $\endgroup$
    – Glen_b
    Commented Feb 27, 2014 at 20:30
  • $\begingroup$ What information would you need for the word 'best' to no longer be ambiguous? $\endgroup$
    – sharshofski
    Commented Feb 27, 2014 at 20:35
  • $\begingroup$ At the least, a clear indication of what you want to optimize. $\endgroup$
    – Glen_b
    Commented Feb 27, 2014 at 20:37
  • $\begingroup$ Ok. I would like to find a method of comparing samples such that I'm able to detect significant changes in the mean and variance. I'm aware there are many ways to approach this problem, but I'm not aware of all of the ones which are most likely to be useful. I'd be interested in discussing any proposals which seem likely to produce good results, which is why I wrote this post. Is that any more clear? $\endgroup$
    – sharshofski
    Commented Feb 27, 2014 at 20:47
  • $\begingroup$ Also, @ your first point: I am aware more data is required for a distribution with high variance. I want to know if there's a clean way to have high confidence my sample is a good representation of the actual distribution, or if not to therefore know I should increase n. $\endgroup$
    – sharshofski
    Commented Feb 27, 2014 at 20:47

1 Answer 1

Reset to default
1
$\begingroup$

You should look into the literature of control charts and quality control. If changes are occurring abruptly, look into methods for detecting changepoints.

For means (and variances), you could plot running estimates based on some window of observations, maybe with exponentially decreasing weights. Also much useful information in the many comments.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I fully support this answer because control charts should always be the first stop for practitioners in my opinion. However, if we are talking specifically about software performance, my experience is that run times are subexponential (fat-tailed) enough that control charts yield more false positives than useful signals. (This means also that log-transform is not sufficient. I haven't tried log°log, come to think of it, but that sounds like it could kill useful signals too. It's an area I'm passively researching!) $\endgroup$
    – kqr
    Commented Jul 12, 2022 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.