# How to calculate overall rankings/ratings from many pairwise comparisons?

I have a number of categories and a bunch of data about pairwise comparisons between those two categories; how do I learn information about overall rankings/ratings/etc. of all the categories?

Let me provide more detail. I'm trying to learn what the general population is most worried about, by surveying them. I have about 20 categories of potential concerns, call them $A, B, ..., T$. I have picked a random sample of people. For each person in the sample I have done the following. I picked a pair of concerns, e.g., $E$ and $Q$. Then I asked the respondent: which are you more worried about, $E$ or $Q$? They could select one of three possible answers: $E$, $Q$, or "about the same for both". I have about 300 responses. Each respondent got a different, independently random selection of a pair of concerns.

How should I analyze this data, to extract as much information as I can? For instance, can I learn anything about what concerns people are generally most worried about, or least worried about? Can I compute some approximate rankings or ratings or some comparisons? And, of course, I am concerned about computing statistical significance for any conclusions drawn. Any advice?

P.S. Feel free to choose any reasonable model for how people will respond. For instance, you could assume transitivity for each individual (each person has an internal ranking of concerns, and answers the question based on that ranking; so if they would answer that they're more worried about $P$ than $Q$ if asked about $P$ vs $Q$, and would say they're more worried about $Q$ than $R$ if asked about $Q$ vs $R$, then if asked about $P$ vs $R$ they would say they're more worried about $P$ than $R$).

• This is essentially what the Bradley-Terry model was made for. You'll have to make some adjustments to handle the possible outcome "about the same for both". There are, however, whole books written on paired comparisons. – cardinal Oct 6 '11 at 22:00
• Even when individuals all maintain transitive rankings, there may be no such consistency for the population. Note, too, that with 20 categories there are 190 possible pairs, implying about 20% of all pairs haven't even shown up for a head-to-head comparison by anyone. Also, some individual concerns have probably been evaluated by fewer than 8 respondents, creating substantial uncertainty for inference to the population. This all suggests it will be difficult or impossible to check the assumptions that any model must make. – whuber Oct 6 '11 at 22:12
• Internal transitivity is a simplifying assumption we often make for the sake of analysis. It is really not true in practice, but in practice there are much bigger biases for us to worry about. Even assuming internal transitivity is not going to help here, if you only have one rating per respondent. – Jonathan Oct 7 '11 at 18:19
• One other thought is that for all surveys, and particularly a survey about political concerns, unbiased wording is extremely important. I would read some articles by Pew Research or another respected survey institution to see how they present unbiased questions. – Jonathan Oct 7 '11 at 18:21

This highlights the importance of choosing an analysis plan before designing the data collection instrument. The analysis plan needs to drive the survey format, not vice versa. While you have intuitively landed on a discrete choice methodology, which is often very useful in answering these questions, I think the design isn't quite sufficient to answer the question you have.

If you had asked how to design a survey to answer this type of question, I would have told you that "Max Diff" (short for maximum difference) is one of the popular ways to do this. In Max Diff, sets of ~5 concerns would be presented together and the respondent would be asked to choose the most and least important. Then a standard scoring methodology is used to provide a group-level score of the relative importance of the various attributes.

One of the reasons that Max Diff is popular is that making responses on a set of attributes tells you about multiple pairwise comparisons, and thus allows you to avoid asking 190 questions and will help you get a higher survey completion rate.

While many statisticians have spent many hours developing strategies to reduce the number of questions that need to be asked, I know that respondents are still required to respond to a series of sets in order to understand how the importance of each item stands relative to the others. For that reason I think the data you have is not going to be sufficient - a single pairwise comparison for each respondent.

Think of each potential concern as being a chess player. When E and Q are paired together and a judgment is made, then either player E wins, there is a draw, or player Q wins.

How can we determine the relative strength of two players who have never met, and are likely to never have had a common opponent?

This problem is solved in chess by Elo type models; see more recent work by Glickman. http://www.glicko.net/research.html There is no requirement that anywhere near all the possible pairs have been directly evaluated.