Suppose that $k$ experts are each asked to rank a set of $n$ objects in order or preference. Let allow ties in the rankings.

John Kemeny and Laurie Snell in their 1962 year book "Mathematical models in the Social Sciences" propose to solve next problem:

PROJECT $1$. Develop a measure of the reliability of a consensus ranking by $k$ experts. For example, this may be based on the largest possible change that can be brought about by changing the ranking of a single expert. (Attention must be paid to the possibility of multiple consensus rankings.) Prove some theorems concerning the most and least reliable consensuses possible for a given $k$.

The book gives notation for rankings and method for rankings aggregation (i.e. getting one "collective" ranking from many "individuals"). But no answer given for the problem above.

First, I thought about Kendall's $W$ coefficient of concordance, but it looks like it doesn't suit. Any ideas are welcome!

bumped to the homepage by Community 2 days ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

As a measure of reliability of a consensus ranking by $k$ experts you can use the $\tau$-extended by Emond and Mason. For extensive explanations take a look to: http://onlinelibrary.wiley.com/doi/10.1002/mcda.313/full or http://www.sciencedirect.com/science/article/pii/S0377221715008048

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.