I would appreciate suggestions on how I could make some specific statistical claims about the efficacy of changes to an algorithm. My statistics background is pretty minimal (I'm working on it).
I would like to compare two versions of the same basic algorithm, however, one has been tweaked (hopefully improved). These are stochastic, population based optimization algorithms, think Genetic Algorithm, or Particle Swarm Optimization.
For instance, my results may look like this (the specific values, functions may vary of course):
benchmark fn Before After dampened_sine -0.597 -0.721 fn1 69.308 -132.688 fn3 -8457.388 -9000.000 fn4 -0.394 -0.463 fn5 0.153 -0.217 haupt_fn1 9.627 1.000 haupt_fn2 5.291 0.000
Any claims I would be making would be specific to the benchmark functions used. I was thinking of using a paired t-test (before and after the change of the algorithm), but I am not sure if that's the correct approach.
I would run each algorithm on each benchmark function N ( > 30?) times for a fixed number of generations and use the mean of these runs as a result to compare. The populations are generated with random values, I could use the same initial seed for each algorithm, but I am assuming that running each algorithm N times with different random seeds would put them more or less on equal footing.
My goal is to write this up in a paper, and be able to state that any observed changes for this specific set of benchmark functions wasn't just due to luck or randomness (given the stochastic nature of these algorithms) but is statistically significant. I am looking for some guidance on how to do this.
I hope the above is clear, I'd be happy to clarify/reword if it would be helpful.