# I have 12 items to get into a single rank order. Can I get this from having a few hundred people see a set of 10 of the possible pairs of items?

I attempted to run a study where people had to rank order 12 statements from best to worst. However, it was extremely messy and difficult for people to rank that many items. Now, my professor and I have decided instead to show people 2 statements at a time and just rate which one is higher. Since it would be way too much to ask people to make every possibly comparison within the set of 12, we'd give each person a subset, maybe 10 from the full set of possible pairs.

I've tried to look up methods to analyze them, but everything I've checked into is not quite right. Some require each person to have rated the full set, like Borda's method or other voting ranking systems. Others don't seem take into account the within-subjects factor, like the Bradley Terry model. And searching for the issue of ranking usually gives me ways to test whether two groups are different, as opposed to one population. For some of these though, I'm not sure if they test/model can't do what I need or if I just don't understand how it is applied. Eventually I'll have the data in R, so for each participant I'll have a set of 10 pairs and how they ranked each one; I haven't figured out how to apply the models to that, though the BradleyTerry2 package seems closest.

Can anyone help me figure out if this analysis is possible before I run the study? Will I be able to get a single, ordered list of the 12 statements with the method I have sketched out? Thank you in advance!

• There is no perfect Pepsi. There are only perfect Pepsi's. link. If you have demographic differences in the rank-givers, then you might be able to not only give ensemble order, but split it by demographic or by demographic interactions. You should also consider replication, so give the same choice, reversed order, to each ranker. – EngrStudent Apr 25 '16 at 14:41
• It isn't completely clear just what you mean by "analyze them" or "this analysis." What is this "single, ordered list" intended to represent? – whuber Apr 25 '16 at 14:42
• By analyze them, I mean put them into an order and test against the null hypothesis (i.e., that there is no true order and thus the order isn't significantly different from random choosing). E.g., if there were only 3 statements (A, B, and C), and there was no defined order, each would win about 50% of the time against the others. A statistical test of the rank would be non-significant. On the other hand, if there was a true order A > B > C, maybe we'd find that A wins 90% of the time against C and 60% of the time against B, while B wins 70% of the time against C (and the rest are inverse). – Bofstein Apr 30 '16 at 5:48
• I am now working with a statistics professor who studied this in his dissertation and though I still din't understand the procedure behind it, he is helping me analyze it. We are using the Bradley Terry method for the basic rank, and he is working on adding in a within-subjects component. Basically each step is a logistic regression predicting the likelihood of choosing each statement over the next one, and we can look at the confidence intervals of each rank to see if they are statistically different from one another. – Bofstein Apr 30 '16 at 5:54