Suppose I have 100 items and I would like to come up with their ranking based on user preferences. i.e. each item should receive a rank from between 1-100, where 1 is the most preferred item and 100 is the least preferred item. Data is collected through pairwise comparisons, where users choose which item they prefer (boolean). Out of these comparison a final ranking will be determined. How do I rigorously determine the number of pairwise comparisons I need per item, to produce a "statistically significant" ranking? I am less interested in how the ranking is constructed and more interested in determining a suitable sample size, but I understand these two may be linked.

  • $\begingroup$ This is a spectacular question. But I have some questions and one concern (and they probably won't fit in one comment...so I'll break them apart). First, ¿are you assuming that the population has some existing count/ranking of the 100 items? If so, ¿how are you envisioning this ranking/counting? For example, you could have a ranking by how many people chose each option for the 1st, 2nd, ... 100th place (a large matrix). Or, you could just focus on what % of people choose each option 1st (or in the top 3 or top 10 etc). $\endgroup$
    – Gregg H
    Commented Jul 4, 2023 at 14:50
  • $\begingroup$ The second question is about what you are trying to capture form the samples. In particular, ¿do you want to capture the rankings or the multinomial probabilities for being first for each item or the population ranked voting matrix? $\endgroup$
    – Gregg H
    Commented Jul 4, 2023 at 14:51
  • $\begingroup$ And for my concern, there is something from voting theory called The Independence of Irrelevant Alternatives (IIA) criteria. In brief, a pairwise comparison like this is subject to violating this criteria. The basic idea is that if you have one item ranked above the others, then it should stay above the others if an irrelevant alternative is introduced. The problem here is that as your # of items gets larger...the number of possible IIAs increases the chances of violating this. $\endgroup$
    – Gregg H
    Commented Jul 4, 2023 at 14:54
  • $\begingroup$ @GreggH Thanks for your questions. Yes I am assuming that there is an intrinsic rank of each of the 100 items. So if you had an infinite number of comparisons then one would find the "true" ranking. I guess one can think as it as, the item with rank 1 is the one that most people agree is better than all the others. In terms of how I will construct the final ranking, I will probably use something like the ELO rating or rank centrality. I am only interested in how close the final ranking is to the "true ranking". $\endgroup$
    – phil
    Commented Jul 13, 2023 at 13:34
  • 1
    $\begingroup$ Can you exclude ties that leave no winners, that is, for a hypothetical example with three items and three users, you get rankings A>B>C, B>C>A and C>A>B? $\endgroup$
    – Ute
    Commented Jul 16, 2023 at 19:13


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.