# How to compare the results of two benchmarks?

I am writing a program and would be interested in knowing how much faster does a version run compared to another.

For that, I wrote a benchmark script that times the run time of my program, repeats the process multiple times and computes the average of the results.

Let's say I run it on the version 1 of my program, I get the value X1. Then I run it on the version 2 and get the value X2.

How can I tell if the difference between the two results is significant or not? In addition, how can I objectively know how many times I need to run the benchmarks to get useful results?

-
–  user10525 Jul 27 '12 at 12:14
When making a statistical comparison you usually need some idea about the spread of the measured quantities. Are you able to extract the run times from each repetition, so that the spread can be investigated? –  MånsT Jul 27 '12 at 12:14
Yes I can retrieve the run time of each repetition. –  Thomas Jul 27 '12 at 12:16
How many repetitions are you able to do within resonable time? 10? 100? 10 million? –  MånsT Jul 27 '12 at 12:20
I'm recreating two data sets and try your answer with R. I will keep you posted! –  Thomas Jul 27 '12 at 13:11