Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am writing a program and would be interested in knowing how much faster does a version run compared to another.

For that, I wrote a benchmark script that times the run time of my program, repeats the process multiple times and computes the average of the results.

Let's say I run it on the version 1 of my program, I get the value X1. Then I run it on the version 2 and get the value X2.

How can I tell if the difference between the two results is significant or not? In addition, how can I objectively know how many times I need to run the benchmarks to get useful results?

share|improve this question
1  
    
When making a statistical comparison you usually need some idea about the spread of the measured quantities. Are you able to extract the run times from each repetition, so that the spread can be investigated? –  MånsT Jul 27 '12 at 12:14
    
Yes I can retrieve the run time of each repetition. –  Thomas Jul 27 '12 at 12:16
    
How many repetitions are you able to do within resonable time? 10? 100? 10 million? –  MånsT Jul 27 '12 at 12:20
1  
I'm recreating two data sets and try your answer with R. I will keep you posted! –  Thomas Jul 27 '12 at 13:11
show 7 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.