6
$\begingroup$

I'm doing timings of an application which speeds up as we add threads. However, the times vary on repeated runs. What I'd like to have is a speedup chart which shows how many times faster than the baseline the application is, for each thread count I test on.

The data I was planning on gathering was 5-10 runs for each number of threads, from which I can get standard deviation for the times at each thread count. But I'm at a loss as to how to represent this deviation in terms of 1 std.dev error bars on a speedup-curve.

I'm thinking deviation I want to show is the std. of the speedup calcs, not the times intially right?

To make this more concrete:

  • Suppose as a baseline, we have the times 10s,10s,10.1s,9.9s (sd = 0.081);
  • at 4 threads, we have 2.5,2.5,8,2.8 (sd = 2.72);
  • Speedups are 4,4,1.2625,3.5357 (sd = 1.3)

So the if I'm plotting the point for 4 threads, does it make sense to have the 1 std. error bar be 1.3 (2.6 in both dir)? ( ignoring skew of course. )

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I would be more concerned about the apparent large deviation from 2.5 to 8 for a single thread. Was this due to some other factor that isn't code dependent, e.g. memory dump or another program using the core at the same time? $\endgroup$
    – adunaic
    Commented Oct 10, 2012 at 13:28
  • $\begingroup$ I would do it on a log scale. $\endgroup$
    – Mike Dunlavey
    Commented Feb 7, 2013 at 16:00

2 Answers 2

Reset to default
2
$\begingroup$

So, further research on this topic has led me to conclude that the correct way of doing this is going to involve Fieller's Theorem, which is for constructing the confidence interval of the ratio of two means --- a speedup ratio!

I've not completely worked this out, but for future people trying to figure this out, I'm hoping it will serve as a pointer.

This is the paper which set me on the right path, though, I am not entirely convinced of their methodology.

Improve this answer
$\endgroup$
0
$\begingroup$

Have you considered using candlesticks over the top of a trendline?

The candlestick body could be placed on each thread interval, and expanded in height a certain number of pixels per unit of the standard deviation. The relative size differences of the candlestick bodies would then demonstrate the change in deviation from one interval to the next.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.