Paired Comparison Testing for Best Color Suppose I have a set of 5 colors and 3 judges whose job is to choose their favorite color in the set. Each judge will be shown a screen with 2/5 colors and then another new pair of colors from the set and then a new pair until all pairs are exhausted. For each pair, the judge would click and choose one out of the two colors that they like more before moving on to the next pair.
In this case, each judge should see 10 unique pairs of colors (n*(n-1)/2, where n is the number of colors) generated from the set of 5 colors. For a small number of colors this wouldn't take too much time but with, say, 50 colors there would be 1,225 pairs for a judge to click through!!
In the case of having 50 colors, is there a way that I could increase the number of judges to, say, 100 and then only show each judge 20 pairs of colors and then be able to infer the "global" or overall color winner?
What is the statistical test or analysis method that I am describing?
 A: Not a full answer but some potentially useful comments:
This kind of experimental design is also known as paired comparisons and a popular and simple stochastic model for it is the Bradley-Terry model. This tries to recover a latent strength for each object in the comparison by fitting a logistic regression model. It is also applicable if the subjects that do the rating do not see all $n \choose 2$ possible pairs. However, I'm not sure whether there are results how you have to design the experiment in order to get the full vector of all $n$ strenghts if you limit the number of comparisons for each subject. As the model is a simple logistic regression there may be results for GLMs that could be leveraged, though.
R packages implementing the Bradley-Terry and related models include: prefmod (http://dx.doi.org/10.18637/jss.v048.i10), BradleyTerry2 (http://dx.doi.org/10.18637/jss.v048.i09), eba, psychotools, among others.
As for the application to color preferences: I would also doubt that you will get a winner (= color with the highest strength) whose strength significantly differs from the second- or third-ranked color in a design with 50 colors and only 100 judges. Also, one would have to keep in mind that the judges' preferences may vary or depend on further covariates etc.
