I want to compare performance of two stochastic optimization algorithms: A1 and A2. Performance is defined as the output of a run. The lower the output is, the better is the solution the algorithm found. There are 15 problem instances I'm supposed to solve. However, I cannot run algorithms just once per instance, since they are non deterministic and don't always return the same result.
I think I could compare them as follows:
Run A1 and A2 for N times per instance. Conduct parametric or non parametric independent samples test, depending on normality and variances of the distributions, to test the following hypotheses:
H0: mean(performance of A1) = mean(performance of A2)
H1: mean(performance of A1) =/= mean(performance of A2)
After repeating this procedure for each instance, I would report for which I have found significant difference between A1 and A2 and end there.
However, I'm concerned with whether this approach is suitable. In the literature, statistical tests are rarely used. My questions are as follows:
- Could I consider the outputs for A1 and A2 to be independent samples? I cannot come up with any criterion for pairing, so I guess the samples are independent, no matter that the algorithms are run on same instances.
- If this is not a good approach, which statistical test should I use? I'm rather new to statistics, so any help will be very appreciated. :)
Thanks in advance. :)