Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am writing a program and would be interested in knowing how much faster does a version run compared to another.

For that, I wrote a benchmark script that times the run time of my program, repeats the process multiple times and computes the average of the results.

Let's say I run it on the version 1 of my program, I get the value X1. Then I run it on the version 2 and get the value X2.

How can I tell if the difference between the two results is significant or not? In addition, how can I objectively know how many times I need to run the benchmarks to get useful results?

share|improve this question
When making a statistical comparison you usually need some idea about the spread of the measured quantities. Are you able to extract the run times from each repetition, so that the spread can be investigated? – MånsT Jul 27 '12 at 12:14
Yes I can retrieve the run time of each repetition. – Thomas Jul 27 '12 at 12:16
How many repetitions are you able to do within resonable time? 10? 100? 10 million? – MånsT Jul 27 '12 at 12:20
I'm recreating two data sets and try your answer with R. I will keep you posted! – Thomas Jul 27 '12 at 13:11

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.