# How to supplement comparing benchmark results (means) with a two-sided t-test?

When comparing two runs (means) of different benchmark tests I use an unpaired two-sample t-test assuming equal variance.

The problem I've run into is this often results in the same benchmark test result being statistically significantly different from re-runs of the same benchmark test on the same computer/device. I noticed that from run to rerun the difference can swing about ~5%. What is the correct way to account/allow for this?

## Update (1.22.2012 @10pm)

Here is a link to a test page that will run the same benchmark 30 times (please be patient) and then create a histogram of the mean values represented as a percent of the most frequent mean value (lowest mean value to the far-left, highest to the far right).

Each one of the 30 benchmark runs repeats the test, collecting samples, as much as possible for 5 seconds. So it takes about 2 ½ minutes to finish all 30 runs.

In this example the sample size for each of the 30 benchmark runs in Chrome will range from something like 86 to 93 and in Mobile Safari something like 47 to 53.

Even though the results are displayed in operations per second the samples are composed of the time, in seconds, it takes to execute a test one time. This results in mean values that are very small. In this example, in Chrome, they are as small as 0.0001135 and 0.0001137 and 0.0001128.

When the 30 benchmarks are finished the result cells are highlighted. The green colored result cells are those that are the fastest and statistically indistinguishable from each other (if this were perfect all cells would be green).

Here are some screenshots of the histogram from Chrome and Mobile Safari:

The distribution is inconsistent from test to re-test and from Chrome to Mobile Safari. This may be because the means are so small. I currently clip the mean values, for the histogram, to 7 decimal places.

## Update (1.23.2012 @2:30am)

Here is some example data that would be used to generate a histogram:

{
// "the mean": frequency the mean was computed
"0.0001465":1,
"0.0001562":1,
"0.0001564":1,
...
}


When I clip the means to 5 decimal places and rerun I get data like:

Chrome #3

{ "0.00012": 30 }


All 30 tests have the same mean when clipped to 5 decimal places.

## Update (1.25.2012 @12:40am)

In my update from 1.22.2012 I mentioned that each benchmark run collects samples for 5 seconds. Each sample is created by executing the test as many times as possible in ~50 milliseconds. Timer resolution in JS in 1ms so to get measurement uncertainty of ~1% I (1/2)/0.01 = 50). After it's clocked I add the computed time per call (timeTaken / numberOfCalls) to the sample and repeat the process collecting samples for 5 seconds.

-
Sounds as if you have confirmed that the same device will produce very different sets of numbers. What else are you looking to find out? (What would "accounting for" the variability mean?) –  rolando2 Jan 22 '12 at 17:50
What sort of sample size do you have in the benchmarks? If the sample size is very small, and the distribution of benchmark results is far from Normal, the t-test may not work well. –  jbowman Jan 22 '12 at 19:27
Shouldn't re-runs on the same machine contribute to within-sample variance rather than between-sample variance? I don't do machine benchmarking so I may be missing something with this comment. –  Michelle Jan 22 '12 at 23:13
@jbowman The sample size can be very high depending on the code being benchmarked. Each benchmark will repeat a test as many times as possible in 5 seconds. So the sample size in Chrome may be something like 86 to 93. –  John-David Dalton Jan 23 '12 at 3:26
Er, if the means are as small as you say is it possible that an accumulation of floating point errors accounts for your differences? –  rpierce Feb 18 '13 at 18:31

I would like to assume that the tests are reasonably close, in this case identical tests seem to fall within ~5% of each others means. But the t-test is too picky with what it considers statistically significant when comparing benchmark results. Is there another test I can use that is less sensitive to the differences in mean value? Is the fact that the mean values are so small, something like 0.0001135, a problem? –  John-David Dalton Jan 23 '12 at 3:29
@John-DavidDalton: Statistical tests are insensitive to whether something is practically significant. They only are sensitive to what is statistically significant. For example, tests of normality frequently report non-normality with large sample sizes even when the underlying distribution is practically normal. You have two possible solutions to adjusting how sensitive the t-test is. One is to adjust the $\alpha$ value. E.g., you could arbitrarily decide that only p < 9.865876e-10 is statistically significant. Two is to round to a lower number of decimal places. –  rpierce Feb 18 '13 at 18:34