3
$\begingroup$

I have 3 different heuristics (H1, H2, H3) that are variation of genetic algorithms combined with some other techniques.

The problem that I try to solve is The problem to minimize energy costs for public transportation mean.
So to test the performance of the heuristics , I did generate randomly about 1,000 test instances that consist on passenger demands for this public tranpsort service On those instances, I did run the 3 heuristics to get for each algorithm on each instance the cost of the solution using a fixed seed.

I have as a final data the founded gap (difference from a lower bound) for each algorithm on each instance(about 1000 gap for each algorithm). By analyzing only the mean of overall gaps for each heuristic, it was clear that the H3 is better as it get the lower mean gap.

I want to know what statistical test could I make to verify that there is any real difference between those 3 algorithms (that it does not happen only by chance) and if it would be that H3 is really the best algorithm to get the lower gap?

Thanks.

$\endgroup$
2
  • 1
    $\begingroup$ It may be helpful to provide more context on your study: what are those heuristics, what are the data? $\endgroup$
    – chl
    Dec 12, 2012 at 22:40
  • 1
    $\begingroup$ I did re-edit the question. I hope it is more clear now. $\endgroup$
    – Shiniga
    Dec 13, 2012 at 10:56

1 Answer 1

3
$\begingroup$

I'm slightly out of my depth on the details of statistical testing, but I think an appropriate method here would be to do pairwise comparisons between the algorithms using a Wilcoxon signed-rank test. This should let you say whether the difference between H1 and H2, H1 and H3, or H2 and H3 respectively are consistent with random chance. Like I said though, I'm open to being corrected by someone with a bit more expertise.

You should be somewhat careful with the interpretation of the results. Hypothesis testing is notorious for being misused in several ways, but despite those caveats, I think it's still the most commonly accepted method for verifying results in the metaheuristics literature.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.