I work for an organization that oversees around 180 schools across the country. We regularly collect outcome data from these schools and rank them. I have been asked to do a survey of the national office (around 100 people) to determine what the best and worst perceived schools are, based on the staff's non-data based impression of school quality.

Some additional information:

  • Not all staff members are familiar with all schools. When asked to rank a school "I don't know anything about this school" needs to be an option.
  • Some staff work with a specific subset of schools, so these staff member's knowledge is great for these specific schools.

My first approach would be to randomly generate sets of 10 or so schools, with each school being in 6 or so sets. Then ask each staff member to rank each set, indicating which schools are unknown. I could then average the ranking for each school across it's sets and use that as the global ranking.

Is the above described method viable? Is there a better way? If this way would work, how many data points do I need for each school for the aggregated ranking to be representative?

  • $\begingroup$ Isn't this exactly the Girl comparison in the Facebook legend, and movie? Would the chess ranking algorithm work here with a series of random pairings? $\endgroup$ – user13894 Sep 6 '12 at 22:16

You could use a Bradley-Terry-Luce type model based on pairwise comparisons. Randomly (or otherwise) generate a bunch of pairs of schools and have each staff member look at several pairs and tell you which in the pair is better (or an "I don't know" if they have no familiarity with one or both schools). Then plug this data into the model to get the ranking.

There is a BradleyTerry2 package for R that fits these models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.