I have been working with simulations, among them scheduling simulations. Each simulation chooses the number of steps, the arrival rate of new jobs and the duration of each step for each job. Thus, each simulation is an instance of a problem.
I would like to compare various algorithms for scheduling these instances, so I measure some global metric for each algorithm for each simulation. I end up with data like this
sim alg1 alg2 alg3 alg4 1 5.4 3.5 6.7 5.6 2 10.5 6.8 9.6 9.8 3 8.9 13.4 11.7 12.3 4 8.0 3.4 2.5 6.5 5 9.4 10.3 12.8 9.5
and so on. I usually have less than 10 different algorithms and hundreds of simulations.
I have been using Tukey HSD to analyse the data but I know I am losing power because Tukey does not know that each entry in the different groups are paired.
Is there a multiple comparison procedure to analyze paired data?