I’m working on improving the robustness of a software engineering process that measures performance of programming language compiler and standard library, a.k.a. benchmarks — in the computing sense of the word. The old process used the mean over 1 second periods, probably to deal with outliers that you can see in the raw data: Scatter plot and histogram of benchmark Calculator, d10 series. These outliers are caused by uncontrolled variable-system load.

My aim is to improve the process to be robust in the presence of varying system load. This question is about using statistics to improve the quality of measurements that will then be used for hypothesis testing (did the performance change?).

There are multiple sources of measurement errors (hardware interrupts, context switches, etc.) that inflate the results. With the old process (i.e. averaging per second), these errors were were causing frequent false reports of change.

Since the distribution of errors and resulting data is clearly not normal, I’ve concluded that the mean was the wrong statistic to use for characterizing the measured values.

Complete bechmark suite measurements:

Now I’m a bit stuck answering questions about typical value and its uncertainty. My initial take was to use Median and Interquartile Range. It works much better than Mean and SD in the presence of outliers. But it still varies with the system load quite a lot. I haven’t looked at Mid-Mean and Trimmed Mean in much detail yet.

Looking at histograms, the measurements with a lot of samples have clearly defined Mode (sometimes more than one) that appears to be very robust to the varying system load. But I haven’t found very solid advice on how to pick the mode algorithmically and how to properly characterize the uncertainty of the mode. I’ve read some notes about picking the right bin size and using that for uncertainty.

Some colleagues have argued for use of Minimum, which has some precedent in the literature and appears to be robust in most cases. But I have found few benchmarks in our test suite where its use would be quite problematic. I also don’t know how to characterize the uncertainty if I just picked the most extreme value from a sample...

Most recently I’ve modified the measurement process to report quantiles from the sample (Type R-3, SAS-2). I thought about combining ventiles (20-Quantiles) from several independent measurements to better characterize the sample distribution and use that as input for the next step of hypothesis testing. Would that be more reasonable approach than picking a single typical value and computing its uncertainty, given that our sample distributions are non-normal, with very long tails and can sometimes even be multi-modal?

During exploratory data analysis of raw measurements guided by NIST Engineering Statistics Handbook, many graphical techniques were hindered by the presence of extreme outliers, which dominated the charts. I’ve adapted the technique from box plot to exclude outliers bigger than top inner fence: TIF = Q3 + 1.5 * IQR. This helped with scaling the charts so that main body of the signal became visible.

Given that our data set consists of repeated measurements of the same benchmark program, I believe it’s justified and necessary to remove the extreme outliers, based on the on following advice in the NIST Handbook, Chapter on Statistical control of the measurement process (emphasis mine):

Causes that do not warrant corrective action (but which do require that the current measurement be discarded) are:

  • Chance failure where the precision is actually in control
  • Glitch in setting up or operating the measurement process
  • Error in recording of data

If I use the above technique to preprocess the data, should I take into account this noise filtering step when expressing the confidence intervals for the statistics computed on the cleaned dataset?

  • 1
    $\begingroup$ What statistics are you interested in computing? Why do you think that computing these statistics on the raw values is worse than some modified values? Have you considered that preprocessing data to remove noise (variance) is directly related to a concomitant increase in bias? Or, stated another way, why would it be a bad idea to report a statistic and an estimate of its variability, since a point estimate from noisy data will necessarily have a larger estimate of variability? $\endgroup$ – Sycorax Sep 4 '18 at 19:31
  • $\begingroup$ @Sycorax It might help to look at the linked interactive chart. My initial aim with that analysis was to find a better typical value. Original system was reporting mean and sd, which I think was misguided, because the distribution isn’t normal. So I thought Q1, Q3 and IQR would be better statistics to use. But I know the outliers on the right tail are corrupted samples, caused by uncontrolled variable (system load). I thought that computing typical range would also be a useful statistic, so I’ve devised a naive filter. $\endgroup$ – Palimondo Sep 4 '18 at 20:03
  • 1
    $\begingroup$ Why do you need to report statistics? What bad thing happens if you report the mean? This sounds a lot like an XY problem. $\endgroup$ – Sycorax Sep 4 '18 at 20:13
  • $\begingroup$ @Sycorax I’m not sure what I need. The end goal is to be able to reliably detect statistically significant changes in the measured benchmarks. The mean used by the old system was constantly reporting false improvements and regressions due to the accumulated error from the uncontrolled system load. I’m looking for more robust method to characterize the sample, for the purpose of comparing new and old system for the significant change. $\endgroup$ – Palimondo Sep 4 '18 at 20:23
  • 1
    $\begingroup$ Palimondo, please address Sycorax's points by editing the question (not everyone reads comments, and they shouldn't have to to understand what you mean). $\endgroup$ – mkt Sep 6 '18 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.