Trying to figure out which statistical method to use We have interviews coming for residency positions very soon. We use a 50 point scoring system and each application is interviewed by 5-6 faculty members. Some faculty give points liberally (with scores reaching 40s most of the times) while some are very stringent (with scores in 25s most of the times). The problem we have is that at times, we don't have the same faculty members interviewing all the applicants. In that case, how do I get to match their scores and sort the candidates out using a cumulative score?
Example: Applicant A is interviewed by John and Mark - both liberal interviewers and got a total score of 90.
Applicant B is interviewed by Paul and Thomas on a different day - both stringent interviewers and got a total score of 60. But applicant B is better of the two and if he interviewed on the same day as applicant A, he would have got 95.
So, when we try to arrange the candidates in a descending order, applicant B will have a significant disadvantage. How do I come up with a rating system where the interviewers' scoring can become comparable?
 A: First, I hope that this score is not the only criterion used for ranking residency applicants. When I was on a residency admissions committee, scores from interviews with faculty were only one part of the process. We would then meet together to go down the list carefully, reviewing all the candidates and frequently reordering before submitting the final rank list. Otherwise you are doing the candidates, your program, and your specialty a potential disservice.
Second, you can start getting scores to agree better among interviewers by standardizing each interviewer's scores to a mean of 0 and a standard deviation of 1. For each interviewer, subtract the mean value of all that interviewer's scores from each score, then divide by the standard deviation among that interviewer's scores. That helps take into account both differences in overall levels of scores among interviewers and differences in how widely interviewers spread their scores across the range. The "average" applicant seen by each faculty member then has a score of 0, with about 2/3 of applicants between -1 and +1. Scores among all faculty members then should be on about the same overall scales.
A related approach would be to use the rank orders among candidates for each interviewer. That could work well if all faculty conduced the same number of interviews, but could lead to problems otherwise as it might be hard to have results on the same scale among faculty.
Neither of those approaches will completely get around the problem of different applicants being interviewed by different faculty, but it should help a lot.
Those approaches should help you get an initial list for further discussion, discussion that might explicitly include the differences among interviewers in scoring.
Third, averaging scores even after standardization doesn't deal with potential candidate-specific bias (positive or negative) by a single interviewer. With a larger admissions committee of about a dozen, we agreed to throw out the top and bottom scores for each applicant to try to minimize that type of problem. That would be harder with your small number of interviewers. In that regard, you might consider having 2 faculty interview each candidate together (same room, same time). That helps make sure that interview results are reported back to the full committee faithfully. Otherwise there's a danger that a single interviewer might have misunderstood (or misstated) some applicant's responses.
Finally, as the comment on the question from @Dave notes, adjustment of scores might not be your biggest problem in selecting a rank order among applicants. Do ask whether your entire process is leading to the best selection of new residents.
