My question deals with how to be able to assert that an "improved" evolutionary algorithm is indeed improved (at least from a statistic point of view) and not just random luck (a concern given the stochastic nature of these algorithms).
Let's assume I am dealing with a standard GA (before) and an "improved" GA (after). And I have a suite of 8 test problems.
I run both both of these algorithms repeatedly, for instance 10 times(?) through each of the 8 test problems and and record how many generations it took to come up with the solution. I would start out with the same initial random population (using same seed).
Would I use a paired t-test for means to verify that any difference (hopefully an improvement) between the averages for each test question would be statistically significant? Should I run these algorithms more than 10 times for each test/pair?
Any pitfalls I should be aware of? I assume I could use this approach for any (evolutionary) algorithm comparison.
Or am I really on the wrong track here? I am basically looking for a way to compare two implementations of an evolutionary algorithm and report on how well one might work compared to the other.