0
$\begingroup$

I'm trying to determine if the results that I have obtained from running a series of benchmarks are statistically significant. i.e. if one benchmark really is faster than the other or if it was just a fluke.

  1. I have 5 different programs.
  2. I run these 5 different programs many times each to obtain a dataset of running times.
  3. I then compute the sample standard deviation and the standard error for the 50th, 95th and 99th percentiles.

I set the error bars at +/- two times the standard error.

If two different programs have overlapping error bars I consider the result to be a fluke (inconclusive). Like this

                                   50    95    99              (ratio)    ops/millisecond
cow_append_preallocate_unrolled 0.001 0.001 0.001   1.000 1.000 1.000   53533 (baseline)
         cow_append_preallocate 0.001 0.001 0.002   1.000 1.000 2.000   46866 (inconclusive!)
              cow_append_direct 0.000 0.001 0.008   0.800 1.000 8.444   40561 (inconclusive!)
               cow_append_slice 0.001 0.001 0.002   1.800 1.857 1.889   29114
              cow_append_spread 0.001 0.001 0.002   2.200 2.143 2.333   26025
                 cow_append_for 0.001 0.002 0.004   3.000 2.429 4.000   19507

When I ran my experiment with a sample size of 1000. I get lots of inconclusive results (error bars touching all over) but when I ran my experiment with a sample size of 1 000 000 I got consistent results. In line with my expectations.

I know that I have a lot of variance in my programs that I cannot control for so I've opted to do it this way but I would very much appreciate if someone could point out flaws in my methodology. I've based my work on this paper. There are lots of warnings on the Internet about comparing overlapping error bars but I can't make up my mind about it...

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ Note that statistical significance is not the same as substantial relevance. If you use a huge sample size (as you do), you will get statistical significance even for very small differences that may not be substantially relevant. Chances are you are aware of this, as you actually can see the averages and intervals, so you should know whether the observed differences are meaningful, be they statistically significant or not, but I thought I mention this anyway. $\endgroup$
    – Christian Hennig
    Commented Nov 14, 2021 at 18:10
  • 1
    $\begingroup$ Are the applications of the different programs independent of each other? I'm asking this because I have done benchmarking with certain randomly generated tasks, and then all programs were applied to the same tasks, meaning that results of different programs on the same task are not independent. $\endgroup$
    – Christian Hennig
    Commented Nov 14, 2021 at 18:20
  • $\begingroup$ About this independent thing. I don't understand it. How can it not be independent? I can provide more details if it helps. I have 5 functions that do the exact same thing but in 5 different ways. When I run one function I obtain 1 sample from the infinite population since there's no limit to the number of times I can run the functions. There's no memory involved. The functions are pure functions that run in isolation and only manipulate process memory for the duration of the run. $\endgroup$
    – John Leidegren
    Commented Nov 15, 2021 at 7:34
  • $\begingroup$ I understand now. This warning about statistical significance applies to smaller sample sizes. I this context, what is an interval? Wasn't able to Google it... $\endgroup$
    – John Leidegren
    Commented Nov 15, 2021 at 7:40
  • $\begingroup$ I have 5 functions that do the exact same thing but in 5 different ways On the same or different input? First case gives dependence, second not ... $\endgroup$
    – kjetil b halvorsen ♦
    Commented Nov 15, 2021 at 13:37

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.