How can people with random skill levels rank other skill levels? I sometimes attend a large dance teaching event.
At this event, students are allocated into groups with other students of roughly the same level through 2 reviews - first a "peer review" that sorts students into roughly the correct levels, then a teacher confirmation review where the teachers assess the groups as a whole and move students up or down levels to balance them.
The "peer review" works like this. Each student is given 7 slips of paper with that student's id number printed on them. Each student dances with another student chosen more or less randomly, they exchange 1 slip of paper with that other student, and each student secretly writes their ranking of the other student on the exchanged slip (which has the other student's id number on it). Scores are a number from 1-6, described as 1 meaning "the worst dancer you've ever danced with" and 6 meaning "the world dance champions in this style".
Said another way : Student X holds 7 slips of paper, labelled here X1, X2, ... X7. Each slip has X's id number printed on it. In round 1, X dances with Y. X gives Y slip X1, and Y gives X slip Y1. X now holds slip Y1 (and X2...X7, not relevant for round 1). X writes X's assessment of Y on slip Y1. Y writes Y's assessment of X on slip X1. 
All slips go into a hat. Overnight all the slips (500 students, so 3,500 slips) are typed into a computer program which ranks the students into 10 level groups. 
It seems to work surprisingly well, despite some obvious problems. For example, a beginner student doesn't have a good way of scoring a more advanced student. Some people may score high, others low. Different people will have different spreads of rankings, and so on.
Question : how would one solve this problem, where random skill levels are providing un-moderated scores of other skill levels? How many dances are requried to achieve a reasonable ranking? (they claim that only 3 dances are required).
 A: This is a case of the so-called "Wisdom of Crowds": non-experts pooling their knowledge can perform very well and even outperform experts. The first mention of this involved a crowd of people estimating the weight of an ox at a fair. The average (either the mean or the median) estimated weight was very close to the actual value (and IIRC, better than the estimate of trained butchers). You may be interested in the book.
The key points here are:


*

*People may not be experts, but they can at least give some information, even if it's only a rough ranking of their dance partners.

*Your dancers don't actively sabotage your system. Or if a few do, then their effects are washed out by the large number of honest dancers. If more than half your dancers start returning random answers, the system will break.

*The process is repeated a few times, which will give the algorithm more data to work with and reduce variability.


How many dances are required will of course depend on a few factors:


*

*More than seven rankings per dance would mean that fewer dances are required.

*A larger talent spread would require fewer dances (because it's easier to sort people if they differ more strongly - the signal to noise ratio is larger).

*More experienced dancer raters would mean that fewer dances are required.

*What you mean by "a reasonable ranking" will have an impact.

