I'm doing timings of an application which speeds up as we add threads. However, the times vary on repeated runs. What I'd like to have is a speedup chart which shows how many times faster than the baseline the application is, for each thread count I test on.

The data I was planning on gathering was 5-10 runs for each number of threads, from which I can get standard deviation for the times at each thread count. But I'm at a loss as to how to represent this deviation in terms of 1 std.dev error bars on a speedup-curve.

I'm thinking deviation I want to show is the std. of the speedup calcs, not the times intially right?

To make this more concrete:

• Suppose as a baseline, we have the times 10s,10s,10.1s,9.9s (sd = 0.081);
• at 4 threads, we have 2.5,2.5,8,2.8 (sd = 2.72);
• Speedups are 4,4,1.2625,3.5357 (sd = 1.3)

So the if I'm plotting the point for 4 threads, does it make sense to have the 1 std. error bar be 1.3 (2.6 in both dir)? ( ignoring skew of course. )