Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have seen surveys that use the following method to rank multiple items against each other. Let's say you have 25 items and you want to know their rank order (according to the survey-taker's preference). You pick 5 at random and ask the survey-taker to pick their favourite and least favourite of the group. You can then pick another 5 and repeat the process until the list is exhausted. You can then pick the favourites and rank them against each other and then do the least favourites and then the ones in the middle and through an iterative process, rank the entire list.

I'm guessing this survey method has a name and I was hoping someone could tell me what it is. If you know of a survey site that can implement it, that would be very helpful. Thanks.

share|improve this question
Are you perhaps thinking of MaxDiff, i.e., Best-Worst Scaling? – Karl Ove Hufthammer Feb 2 '14 at 20:36
@KarlOveHufthammer That is it! Thank you. If you want to change your comment to an answer, I'd be happy to award you the correct answer and bounty. – zgall1 Feb 4 '14 at 15:24
up vote 3 down vote accepted

Are you perhaps thinking of MaxDiff, a.k.a., best−worst scaling?

MaxDiff sounds like a wonderful method, but do note that it isn’t as amazing as some make it out to be, and it has inherent problems. These blog posts have some information on this:

share|improve this answer
Thanks for the answer and tips. For my purposes, I think this method will be sufficient. – zgall1 Feb 4 '14 at 20:37

I'm afraid I'm not answering your question exactly, but I am aware of a similar method called Q-sorting. This involves sorting items into a forced distribution according to predefined groups, such as, most/moderately/least true/preferred, and sometimes repeating this sorting process for the middle group to divide it into three intermediate groups (e.g., more/moderately/less). Typically, I think q-sorting forces answers into a quasi-normal distribution, so that the most items get sorted into the middle group, and the fewest items get sorted into the two extreme groups, but the number of items sorted into each group on either side of the middle group is symmetrical (e.g., same number of items sorted into both the most and least groups).

The Riverside Accuracy Project (RAP) has a web page with applied examples of a Q-sorter program that you might find helpful. I imagine others are available as well, and hope to see other answers here! I'm interested in these methods myself, and have designed my own measure of values that forces 25 items into 5 groups of 5 each according to their importance. That is, my own application is different from the quasi-normal forced distribution in that it forces a flat distribution instead.

The basic approach of presenting a limited, randomized subset of items for comparison to one another is not one I used, but I think it's used in the Q-sorting program at the RAP page above. Since different methods don't necessarily force answers into groups of the same or pre-specified sizes, I don't see why you couldn't just vary the method to achieve rankings instead of groupings. Maybe this wouldn't be Q-sorting per se though.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.