I am writing a program and would be interested in knowing how much faster does a version run compared to another.
For that, I wrote a benchmark script that times the run time of my program, repeats the process multiple times and computes the average of the results.
Let's say I run it on the version 1 of my program, I get the value
X1. Then I run it on the version 2 and get the value
How can I tell if the difference between the two results is significant or not? In addition, how can I objectively know how many times I need to run the benchmarks to get useful results?