Suppose I have 100 items and I would like to come up with their ranking based on user preferences. i.e. each item should receive a rank from between 1-100, where 1 is the most preferred item and 100 is the least preferred item. Data is collected through pairwise comparisons, where users choose which item they prefer (boolean). Out of these comparison a final ranking will be determined. How do I rigorously determine the number of pairwise comparisons I need per item, to produce a "statistically significant" ranking? I am less interested in how the ranking is constructed and more interested in determining a suitable sample size, but I understand these two may be linked.
7
-
$\begingroup$ This is a spectacular question. But I have some questions and one concern (and they probably won't fit in one comment...so I'll break them apart). First, ¿are you assuming that the population has some existing count/ranking of the 100 items? If so, ¿how are you envisioning this ranking/counting? For example, you could have a ranking by how many people chose each option for the 1st, 2nd, ... 100th place (a large matrix). Or, you could just focus on what % of people choose each option 1st (or in the top 3 or top 10 etc). $\endgroup$– Gregg HJul 4 at 14:50
-
$\begingroup$ The second question is about what you are trying to capture form the samples. In particular, ¿do you want to capture the rankings or the multinomial probabilities for being first for each item or the population ranked voting matrix? $\endgroup$– Gregg HJul 4 at 14:51
-
$\begingroup$ And for my concern, there is something from voting theory called The Independence of Irrelevant Alternatives (IIA) criteria. In brief, a pairwise comparison like this is subject to violating this criteria. The basic idea is that if you have one item ranked above the others, then it should stay above the others if an irrelevant alternative is introduced. The problem here is that as your # of items gets larger...the number of possible IIAs increases the chances of violating this. $\endgroup$– Gregg HJul 4 at 14:54
-
$\begingroup$ @GreggH Thanks for your questions. Yes I am assuming that there is an intrinsic rank of each of the 100 items. So if you had an infinite number of comparisons then one would find the "true" ranking. I guess one can think as it as, the item with rank 1 is the one that most people agree is better than all the others. In terms of how I will construct the final ranking, I will probably use something like the ELO rating or rank centrality. I am only interested in how close the final ranking is to the "true ranking". $\endgroup$– philJul 13 at 13:34
-
1$\begingroup$ Can you exclude ties that leave no winners, that is, for a hypothetical example with three items and three users, you get rankings A>B>C, B>C>A and C>A>B? $\endgroup$– UteJul 16 at 19:13
|
Show 2 more comments