Is Using Differnet Evaluators for Head-to-Head Competing Grants a Fair Process?

Question

We evaluate around 100 grant proposals each year using a team of 15 evaluators who all use the same criteria. 100 grant proposals is a lot for any one evaluator to do and they are experiencing fatigue. So to lessen the burden we are thinking about dividing up the grants into say four batches and asking a subset of evaluators to evaluate just one batch and not all 100. We would then compare all 100 project scores as this is a head-to-head competition. My concern is that it seems inappropriate to rank these proposals together given they were evaluated by different people. Is that a rational concern? What else should we be thinking about as flaws in this system? Or am I wrong and this is an accepted practice without any statistical or qualitative pitfalls?

Any thoughts are much appreciated.

Answer 1

I agree there are theoretical risks to strategies that don't have everyone reading everything. But having had considerable personal experience managing and serving on review panels, I can say there are also risks from overloading reviewers.

In part, reasonable solutions depend on (1) whether it is natural to divide applications into subfields and (2) whether you really need to rank all applications, or just enough to determine which get awards (in case, say there is funding for only 20% to be funded).

Initially stratify by field: You could divide proposals into $g$ groups, and choose some reviewers in relevant subfields for each. Then have everyone (or a super-panel) pick among the top-rated proposals in each subfield. (Ideally at this final step, everyone will judge among at twice as many proposals as can be funded.)

It is probably best for the intellectual health of the general area that funding has a clear chance to be spread across sub-fields, and this strategy works toward making that possible.

Initially stratify by track records of reviewers: Inevitably, some reviewers are tougher than others and some are better at spotting promising research than others. To the extent that current reviewers have served before, look at past performance and try to rate them (if only roughly and qualitatively) on both scales. Then make sure sub-panels have a balance of reviewers. Again here, a larger group picks among the proposals top-rated by sub-panels.

(Candidly, this can only work if it is administratively and legally feasible for reviewer ratings to be kept confidential.)

Statistical design strategies: You might look into design structures such as 'balanced incomplete block designs' and 'partially-balanced incomplete designs'. Then as far as allowed by the design, assign proposals to blocks at random. Such designs can assure that proposals are read by overlapping 'blocks' of reviewers, so that there is some basis at the end for overall ranking. Statistical ranking, rather than a final larger panel, determines final awards.

This may sound great, and it is certainly better than nothing. But my personal experience is that initial panels based on subfields and/or balanced by reviewer track record have given what I felt were fairer results.

Note: It might be interesting to get answers to this question on out sister 'Academia' site.