Reducing a pool of 175 applicants to a list of 20 applicants from 3-factor evaluations I’m chairing a search committee for an academic position. We have a pool of 175 candidates and a committee of 5 who will rank each candidate in three different factors, using scores from 1(lowest)–4(highest). I’ve encouraged the evaluators to use more 2s and 3s and fewer 1s and 4s. When the evaluators finish their work, I’d like to get the pool of 175 down to list of 20–25. I’d like to ensure that these have high scores across all evaluators, and I also want to ensure that each evaluator makes similarly substantial contributions to the list of 20–25. Lastly, I want to account for different standard deviations across evaluators. I plan on working in R and have an intermediate grasp on statistics. Are there statistical techniques that would be considered best practices in this situation?
 A: What about something like this?
n = 175 # number of applicants

# generate dummy marks 
marks = data.frame(id = c(seq(1,175)), c1 = round(runif(n)*3+1,0), c2 = round(runif(n)*3+1,0), c3 = round(runif(n)*3+1,0),
                       c4 = round(runif(n)*3+1,0), c5 = round(runif(n)*3+1,0))
# compute mean per candidate
marks$mean = (marks$c1 + marks$c2 + marks$c3 + marks$c4 + marks$c5)/5

# compute sum per candidate 
marks$sum = marks$c1 + marks$c2 + marks$c3 + marks$c4 + marks$c5

# compute sd per candidate 
marks$sd = sd(c(marks$c1, marks$c2, marks$c3, marks$c4, marks$c5))

#order data
ranking = marks[order(marks[,"mean"], marks[,"sum"], marks[,"sd"], marks[,"c1"],marks[,"c2"], marks[,"c3"],
                      marks[,"c4"], marks[,"c5"], decreasing = TRUE ), ]

With the last line you order your data-frame according to multiple columns. 
At the moment I set the mean assessment as the one with highest priority.
To be sure that you give the same importance to all the committee members you could order the data-frame multiple times permuting the order of the committee members and every time you select the best 25 candidates.
For the final decision you take the 25 who appear the highest number of times.
