Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite
2

I have a number of categories and a bunch of data about pairwise comparisons between those two categories; how do I learn information about overall rankings/ratings/etc. of all the categories?

Let me provide more detail. I'm trying to learn what the general population is most worried about, by surveying them. I have about 20 categories of potential concerns, call them $A, B, ..., T$. I have picked a random sample of people. For each person in the sample I have done the following. I picked a pair of concerns, e.g., $E$ and $Q$. Then I asked the respondent: which are you more worried about, $E$ or $Q$? They could select one of three possible answers: $E$, $Q$, or "about the same for both". I have about 300 responses. Each respondent got a different, independently random selection of a pair of concerns.

How should I analyze this data, to extract as much information as I can? For instance, can I learn anything about what concerns people are generally most worried about, or least worried about? Can I compute some approximate rankings or ratings or some comparisons? And, of course, I am concerned about computing statistical significance for any conclusions drawn. Any advice?

P.S. Feel free to choose any reasonable model for how people will respond. For instance, you could assume transitivity for each individual (each person has an internal ranking of concerns, and answers the question based on that ranking; so if they would answer that they're more worried about $P$ than $Q$ if asked about $P$ vs $Q$, and would say they're more worried about $Q$ than $R$ if asked about $Q$ vs $R$, then if asked about $P$ vs $R$ they would say they're more worried about $P$ than $R$).

share|improve this question
4  
This is essentially what the Bradley-Terry model was made for. You'll have to make some adjustments to handle the possible outcome "about the same for both". There are, however, whole books written on paired comparisons. – cardinal Oct 6 '11 at 22:00
3  
Even when individuals all maintain transitive rankings, there may be no such consistency for the population. Note, too, that with 20 categories there are 190 possible pairs, implying about 20% of all pairs haven't even shown up for a head-to-head comparison by anyone. Also, some individual concerns have probably been evaluated by fewer than 8 respondents, creating substantial uncertainty for inference to the population. This all suggests it will be difficult or impossible to check the assumptions that any model must make. – whuber♦ Oct 6 '11 at 22:12
Internal transitivity is a simplifying assumption we often make for the sake of analysis. It is really not true in practice, but in practice there are much bigger biases for us to worry about. Even assuming internal transitivity is not going to help here, if you only have one rating per respondent. – Jonathan Oct 7 '11 at 18:19
1  
One other thought is that for all surveys, and particularly a survey about political concerns, unbiased wording is extremely important. I would read some articles by Pew Research or another respected survey institution to see how they present unbiased questions. – Jonathan Oct 7 '11 at 18:21

2 Answers

up vote 2 down vote accepted

This highlights the importance of choosing an analysis plan before designing the data collection instrument. The analysis plan needs to drive the survey format, not vice versa. While you have intuitively landed on a discrete choice methodology, which is often very useful in answering these questions, I think the design isn't quite sufficient to answer the question you have.

If you had asked how to design a survey to answer this type of question, I would have told you that "Max Diff" (short for maximum difference) is one of the popular ways to do this. In Max Diff, sets of ~5 concerns would be presented together and the respondent would be asked to choose the most and least important. Then a standard scoring methodology is used to provide a group-level score of the relative importance of the various attributes.

One of the reasons that Max Diff is popular is that making responses on a set of attributes tells you about multiple pairwise comparisons, and thus allows you to avoid asking 190 questions and will help you get a higher survey completion rate.

While many statisticians have spent many hours developing strategies to reduce the number of questions that need to be asked, I know that respondents are still required to respond to a series of sets in order to understand how the importance of each item stands relative to the others. For that reason I think the data you have is not going to be sufficient - a single pairwise comparison for each respondent.

share|improve this answer
up vote 2 down vote

Think of each potential concern as being a chess player. When E and Q are paired together and a judgment is made, then either player E wins, there is a draw, or player Q wins.

How can we determine the relative strength of two players who have never met, and are likely to never have had a common opponent?

This problem is solved in chess by Elo type models; see more recent work by Glickman. http://www.glicko.net/research.html There is no requirement that anywhere near all the possible pairs have been directly evaluated.

You can also just start with the Wikipedia page, which isn't bad. http://en.wikipedia.org/wiki/Elo_rating_system

The Elo model also appears in the movie Social Network -- it's the equation scribbled on the window that is used to determine the relative hotness of various women.

share|improve this answer
In chess, a ranking system makes sense because the outcome of each game depends only on the players and the purpose is to estimate the strengths of the players. Here, the outcomes depend on 300 different people and an implicit objective is to estimate how a population of people would rank the concerns overall. Thus, although one could apply the ELO models to the current question, they don't appear to be suitable. – whuber♦ Oct 7 '11 at 14:51
Based on the Wikipedia page provided in the link in the question, it appears that the Elo ratings model is essentially identical to the Bradley-Terry model using a fairly natural modification to handle draws. However, it appears to use a quite (statistically) inefficient means for estimating the players' ratings. – cardinal Oct 7 '11 at 17:07
Re "inefficient": Elo models were designed to be calculated by hand by tournament administrators, so that's not surprising. That's why I gave the Glickman reference. There's also a Kaggle contest designed to provide improved predictions in this area, although I haven't reviewed it. kaggle.com/c/chess @whuber: he's got a random sample, so I'm not sure agree. Note similarity to Bradley-Terry model. However, OP should compare my reputation score with whuber's (hint: his is higher) when evaluating my comment. – zbicyclist Oct 10 '11 at 16:23
@zb Randomness is good, but I don't think it's the issue. The question in my mind concerns whether the ELO method truly addresses the question. This is an issue that ought to be discussed on its merits, not on the basis of reputations of those involved :-). – whuber♦ Oct 10 '11 at 16:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.