Background: $N$ research funding proposals are to be ranked to inform funding decisions. Each proposal is assessed and ranked by 2-4 subject matter experts (SMEs) who only see a small number of proposals. Typically $N$ is many hundreds and the SMEs only assess 2-5 proposals. (There is other information as well but the question is just about the utility of the rank information.)

Question: What can be inferred about the 'actual' rank out of $N$ given the proposal's ranks in a few small samples?


If you're interested in use (more than in development), you can try a rating system, as most known Elo, building 'matches' with pair of proposals.

Rankade, our free-to-use ranking system (here's a comparison with aformentioned Elo and more), allows 'matches' with more than two items (eg proposals), as per your needs. So, you can insert a single match for every SME rank (so 2 to 5 faction-proposals per match), and have a single ranking featuring all proposals.

If you manage many hundreds of proposals, probably you need more than 2-4 SME to obtain a significant ranking in this way, but after some 'matches' you can anyway

  • make a pre-shortlisting process, excluding lower scores proposals,
  • refine your rank, specially building 'matches' for proposals with highest and/or similar scores.
  • $\begingroup$ thanks Tomaso - I agree that 2-4 SMEs ranking a proposal with respect to a small sample of other proposals is insufficient to obtain a significant overall ranking - but the question is whether there is any useful information at all in those small sample rankings. My intuition about it is that the error bars are just too large to pay any attention to it at all, but others disagree - so I am hoping to get a clear answer. $\endgroup$ – AMG Aug 31 '16 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.