Suppose that $k$ experts are each asked to rank a set of $n$ objects in order or preference. Let allow ties in the rankings.

John Kemeny and Laurie Snell in their 1962 year book "Mathematical models in the Social Sciences" propose to solve next problem:

PROJECT $1$. Develop a measure of the reliability of a consensus ranking by $k$ experts. For example, this may be based on the largest possible change that can be brought about by changing the ranking of a single expert. (Attention must be paid to the possibility of multiple consensus rankings.) Prove some theorems concerning the most and least reliable consensuses possible for a given $k$.

The book gives notation for rankings and method for rankings aggregation (i.e. getting one "collective" ranking from many "individuals"). But no answer given for the problem above.

First, I thought about Kendall's $W$ coefficient of concordance, but it looks like it doesn't suit. Any ideas are welcome!


As a measure of reliability of a consensus ranking by $k$ experts you can use the $\tau$-extended by Emond and Mason. For extensive explanations take a look to: http://onlinelibrary.wiley.com/doi/10.1002/mcda.313/full or http://www.sciencedirect.com/science/article/pii/S0377221715008048


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.