Choosing the fastest code between two equivalent codes I want to learn how to use statistics to help me choose between two equivalent code. I want to choose the "fastest", the one that would take the less time to execute.
How should I collect data? And what test should I use to help me decide?
Let's take a concrete example, you have two ways to compute the double of a number n:
f(n) = n + n
g(n) = n * 2

How would you collect data? Should you:


*

*Take a random number n, compute f(n) et g(n) 1000 times, recording each time the amount of time it took.

*Take 1000 random numbers, compute f(n) and g(n) for each of those numbers one time.

*Take 1000 random numbers, compute 1000 times f(n) and g(n) for each of those numbers.


(I wrote 1000, but I don't know how many samples would be reasonable)
Now, for that set of data, what test should I choose?
I wrote "fastest" with quotes before, because I don't really know what it means in mathematical terms. Do we want to compare the fastest time between the f set and the g set? Do we want to get the fastest on average? What about errors in the dataset?
With some google search, I saw that the Student T-Test would be reasonable. But I would not know which one to choose, since I don't quite understand the concept of a dependent/independent variable, paired/not paired.
Also I read this answer:
https://stats.stackexchange.com/a/139405/232833
saying that:
"You should not use Student's T-test as it has a lot of assumptions which you can easily and unknowingly violate. It is much better so use some nonparametric tests."
(What are the assumptions that can easily be violated? What nonparametric test to choose?)
I've chosen f and g to understand more about the statistical method to choose the fastest one (so I can apply that with more complex codes/problems). I'm not really interested by programming tips.
 A: You can certainly use a $t$-test since you can (presumably) accumulate as much data as you want, and so the central limit theorem will have had enough time to kick in.  As far as the amount of data that you need, that depends on how much of a difference in runtime you're trying to detect, and how much the runtime varies across independent executions.  You can see this by looking at the definition of the $t$ statistic for two independent samples:
$$
\frac{\bar{x}_1 - \bar{x}_2}{\sqrt{s^2_1 / n_1 + s^2_2 / n_2}} .
$$
The smaller the difference you want to detect ($|\bar{x}_1 - \bar{x}_2|$), or the greater the variability in runtime ($s_1$ and $s_2$), the more data you need ($n_1$ and $n_2$).  Again since you're generating the data yourself, you can probably simulate enough to alleviate these concerns.
I think however that the important questions you want to answer are mostly not statistical in nature.  The time it takes the program to run will also depend on the nature of the inputs as well as the computer architecture used.  Are there certain sets of inputs that you can reasonably expect the program to be given?  What happens when the program is given very large inputs compared to small inputs, and how likely are these input sizes?  Does the winner change as the size of the input changes?  This could happen if (say) you had two programs with runtimes as follows:
$$
T_1(n) = n \\
T_2(n) = 0.001 \cdot n^2 .
$$
If you look at small inputs program one will appear to be slower, even though it is asymptotically faster.  This also shows how you can't necessarily use the conclusion of your experiment to say something about wildly different inputs.  Unless you are fairly sure about the constraints under which the program will run, it might be wise to perform several large experiments under different settings to see if and how the results change.
