How to rank order performance of four systems using aggregates of pairwise assessments of system performance? I am currently designing an experiment to evaluate 4 iterations of a system. As it is rather difficult for a human to judge the output of the system by assigning it a score or several scores, I plan on giving the following task:
Give the human evaluator the outputs of two systems on the same input and ask him to decide which system's output is better in certain regards.
My final desired result is to have a ranked list of system iterations and/or pairwise comparisons stating which system iteration is better.
How should I proceed with the experimental design? What significance test should I use?
Update: It seems that sign test could be appropriate to test whether one system is truly better than the other.
Update2: Does obtaining the ranking of 4 system iterations necessitate performing m trials for each pair or could transitivity be used, as in if A > B and B > C, then A > C, to reduce the number of required trials?
 A: Here's a really basic design for $s=4$ systems. Create a factorial set of trials:


*

*$m$ different inputs

*$p=6  = (s-1)(s)/2= 3(4)/2$ comparisons (i.e., System A with B, A with C, A with D, B with C, B with D, C with D)


Create $pm$ trials and present in random order to $n$ participants.
For each participant, record the proportion of the time that system A, B, C, and D is endorsed as providing better output. You may wish to correct $\alpha$ for the multiple pairwise comparisons.
Report mean proportions for each system to rank order. Complete a repeated measures ANOVA and all pairwise repeated measures t-tests to test for overall and pairwise differences respectively. 
Your statistical power to successfully rank order the systems will be increased by:


*

*increasing the number of trials $pm$ 

*increasing the number of participants $n$

*increasing the performance difference between systems


If $pm$ is too large for any one rater, then you could adopt a procedure for sampling these possible trials. A simple approach would be just to take a random sample from the $pm$ possible trials. Alternatively, you could be more systematic about how you sampled the $pm$ trials. I imagine also that if you wanted to get technical there would be various ways of optimising the efficiency of your sampling. A key consideration is that any sampling procedure does not produce biased estimates.
