3
$\begingroup$

When comparing two runs (means) of different benchmark tests I use an unpaired two-sample t-test assuming equal variance.

The problem I've run into is this often results in the same benchmark test result being statistically significantly different from re-runs of the same benchmark test on the same computer/device. I noticed that from run to rerun the difference can swing about ~5%. What is the correct way to account/allow for this?

Update (1.22.2012 @10pm)

Here is a link to a test page that will run the same benchmark 30 times (please be patient) and then create a histogram of the mean values represented as a percent of the most frequent mean value (lowest mean value to the far-left, highest to the far right).

Each one of the 30 benchmark runs repeats the test, collecting samples, as much as possible for 5 seconds. So it takes about 2 ½ minutes to finish all 30 runs.

In this example the sample size for each of the 30 benchmark runs in Chrome will range from something like 86 to 93 and in Mobile Safari something like 47 to 53.

Even though the results are displayed in operations per second the samples are composed of the time, in seconds, it takes to execute a test one time. This results in mean values that are very small. In this example, in Chrome, they are as small as 0.0001135 and 0.0001137 and 0.0001128.

When the 30 benchmarks are finished the result cells are highlighted. The green colored result cells are those that are the fastest and statistically indistinguishable from each other (if this were perfect all cells would be green).

Here are some screenshots of the histogram from Chrome and Mobile Safari:

Chrome #1, Chrome #2, iPhone results #1, iPhone log #1, iPhone histogram #1, iPhone histogram #2

The distribution is inconsistent from test to re-test and from Chrome to Mobile Safari. This may be because the means are so small. I currently clip the mean values, for the histogram, to 7 decimal places.

Update (1.23.2012 @2:30am)

Here is some example data that would be used to generate a histogram:

{
// "the mean": frequency the mean was computed
  "0.0001465":1,
  "0.0001562":1,
  "0.0001564":1,
  ...
}

When I clip the means to 5 decimal places and rerun I get data like:

Chrome #3

{ "0.00012": 30 }

All 30 tests have the same mean when clipped to 5 decimal places.

Update (1.25.2012 @12:40am)

In my update from 1.22.2012 I mentioned that each benchmark run collects samples for 5 seconds. Each sample is created by executing the test as many times as possible in ~50 milliseconds. Timer resolution in JS in 1ms so to get measurement uncertainty of ~1% I (1/2)/0.01 = 50). After it's clocked I add the computed time per call (timeTaken / numberOfCalls) to the sample and repeat the process collecting samples for 5 seconds.

Improve this question
$\endgroup$
5
  • $\begingroup$ Sounds as if you have confirmed that the same device will produce very different sets of numbers. What else are you looking to find out? (What would "accounting for" the variability mean?) $\endgroup$
    – rolando2
    Commented Jan 22, 2012 at 17:50
  • $\begingroup$ What sort of sample size do you have in the benchmarks? If the sample size is very small, and the distribution of benchmark results is far from Normal, the t-test may not work well. $\endgroup$
    – jbowman
    Commented Jan 22, 2012 at 19:27
  • $\begingroup$ Shouldn't re-runs on the same machine contribute to within-sample variance rather than between-sample variance? I don't do machine benchmarking so I may be missing something with this comment. $\endgroup$
    – Michelle
    Commented Jan 22, 2012 at 23:13
  • $\begingroup$ @jbowman The sample size can be very high depending on the code being benchmarked. Each benchmark will repeat a test as many times as possible in 5 seconds. So the sample size in Chrome may be something like 86 to 93. $\endgroup$
    – John-David Dalton
    Commented Jan 23, 2012 at 3:26
  • $\begingroup$ Er, if the means are as small as you say is it possible that an accumulation of floating point errors accounts for your differences? $\endgroup$
    – russellpierce
    Commented Feb 18, 2013 at 18:31

1 Answer 1

Reset to default
1
$\begingroup$

It seems to me that you have a null hypothesis for the test that relates to the benchmark mean being the same now as it was at some time in the past. Given that (I assume) benchmark results depend on what other things the computer might be doing at the same time, a significant test result with that null hypothesis presents no problems.

If you are simply wanting to produce a reliable benchmark result that can be used for, for example, comparing between computers, then your 'problem' indicates that you probably need to make your sample larger (perhaps with observations scatterd widely over time) or to find out what is the interfering variable and control it.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I would like to assume that the tests are reasonably close, in this case identical tests seem to fall within ~5% of each others means. But the t-test is too picky with what it considers statistically significant when comparing benchmark results. Is there another test I can use that is less sensitive to the differences in mean value? Is the fact that the mean values are so small, something like 0.0001135, a problem? $\endgroup$
    – John-David Dalton
    Commented Jan 23, 2012 at 3:29
  • $\begingroup$ I'm not sure that you are really considering what statistical tests really do, but if you want a test that will deal with a median rather than the mean (medians are less affected by the magnitude of extreme values within the sample), then you could consider a non-parametric test in place of Student's t-test. Try a permutations test on the medians, or a Wilcoxon (Mann-Whitney) test. $\endgroup$
    – Michael Lew
    Commented Jan 23, 2012 at 21:45
  • $\begingroup$ @John-DavidDalton: Statistical tests are insensitive to whether something is practically significant. They only are sensitive to what is statistically significant. For example, tests of normality frequently report non-normality with large sample sizes even when the underlying distribution is practically normal. You have two possible solutions to adjusting how sensitive the t-test is. One is to adjust the $\alpha$ value. E.g., you could arbitrarily decide that only p < 9.865876e-10 is statistically significant. Two is to round to a lower number of decimal places. $\endgroup$
    – russellpierce
    Commented Feb 18, 2013 at 18:34
  • $\begingroup$ For your purposes 7 decimal places picks up variance that is so small you don't care about it. You say that when you round to 5 decimal places all of the values come up the same. Do you care about performance differences in the 6th decimal place? If not, round to 5. Do you care about performance differences in the 5th decimal place, if not, round to 4 ... and so on. I don't quite understand what measurement unit your mean is in... but it doesn't make much sense to record a number of decimal places for your sample that exceeds twice the resolution of your timer. $\endgroup$
    – russellpierce
    Commented Feb 18, 2013 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.