I want to learn how to use statistics to help me choose between two equivalent code. I want to choose the "fastest", the one that would take the less time to execute.
How should I collect data? And what test should I use to help me decide?
Let's take a concrete example, you have two ways to compute the double of a number
f(n) = n + n g(n) = n * 2
How would you collect data? Should you:
- Take a random number n, compute f(n) et g(n) 1000 times, recording each time the amount of time it took.
- Take 1000 random numbers, compute f(n) and g(n) for each of those numbers one time.
- Take 1000 random numbers, compute 1000 times f(n) and g(n) for each of those numbers.
(I wrote 1000, but I don't know how many samples would be reasonable)
Now, for that set of data, what test should I choose?
I wrote "fastest" with quotes before, because I don't really know what it means in mathematical terms. Do we want to compare the fastest time between the
f set and the
g set? Do we want to get the fastest on average? What about errors in the dataset?
With some google search, I saw that the Student T-Test would be reasonable. But I would not know which one to choose, since I don't quite understand the concept of a dependent/independent variable, paired/not paired.
Also I read this answer: https://stats.stackexchange.com/a/139405/232833
saying that: "You should not use Student's T-test as it has a lot of assumptions which you can easily and unknowingly violate. It is much better so use some nonparametric tests."
(What are the assumptions that can easily be violated? What nonparametric test to choose?)
g to understand more about the statistical method to choose the fastest one (so I can apply that with more complex codes/problems). I'm not really interested by programming tips.