Methods for compiler benchmarking? In computer science, statistical knowledge is quite low in general. You rarely see standard deviation in papers. Now I am trying to improve at least my skills, but I find myself utterly confused about the wealth of methods available. Here is my typical use case:
I am working with compilers, so I come up with an optimization, which should make programs faster. For evaluation I have a benchmark suite (e.g. SPEC 2006) with a set of programs. I run it n times with and without my optimization. What should I do now?


*

*Compute sample-means per benchmark program. Gives me speedups per programs. Hopefully, all positive. In this field often small. A 1% improvement is great (for C/C++ compilers on Intel CPUs).

*Compute standard deviation per program. Hopefully, I do not see them overlapping.

*Compute p-value using T-Student. As far as I understood, I should use the "unpaired/independent", "equal variance", "two-tailed" variant. If p-value is high, then I run the benchmark suite again with higher n.


Would it make sense to do more (e.g. confidence intervals)?
How do I correctly summarize the different programs? Mean of means?
Instead of the mean, some people use the best measurement per program. The reason is that a compiler optimization is used once to create a single program, which is evaluated n times. Thus any variance is never an effect of the optimization we want to evaluate. On the other hand you can argue that the optimization can influence the size (strength?) of the variance, which might be interesting to measure.
I never tried a bayesian approach.
 A: You should not use Student's T-test as it has a lot of assumptions which you can easily and unknowingly violate. It is much beter so use some nonparametric tests.
Please read this paper: http://www.jmlr.org/papers/v7/demsar06a.html
It is about comparing classification algorithms, but is analog to your problem.
The only question is, whether you will have independent or dependent (paired) data. If you will compile one source code multiple times on different compilers - then you have independent data. If you have multiple source codes, which you will compile only once on different compilers - then you have dependent (paired) data.
A: Not really an answer to your statistical question, but I think you have an XY problem here.
Unfortunately improvements via compiler optimizations are very difficult to quantify through timings. You will rarely see timing results because there are so many environmental issues that can ruin your measurements (scheduling, context switches, page faults, ...). To reduce the effects of some of those, you most definitely need to abandon user land to do your timings.
Secondly, you don't need timings to quantify improvements in computer science. Look at your assembly code. We know exactly how many cycles each instruction requires. Count the cost of your assembly code compared to unoptimized versions and you are done. No timings, no noise, no statistics, no bias. This would be a far more objective way to quantify improvements made by your optimizations.
