How to combine the grades of different pairs of markers?

Question

I found myself at a difficult situation in deciding how to combine the grades of different pairs of markers such that I can rank the candidates fairly.

I have around 500 candidates and around 30 markers marking 40 candidates selected at random, such that each candidate is evaluated by at least 2 markers since the marking criteria are not fully objective.

Those are the general characteristics of each marker: the minimum, average, and maximum grades they have given and their standard deviation.

Given this, what is the most "fair" way of combining the grades of different (2 or 3) markers?

For each student:

1. Simply average the grades of different markers.
2. Add the Z-Scores of both markers (and ignore the third one, if exists).
3. Add the Min-Max Normalized Scores of both markers (and ignore the third one again, if exists).
4. ?

Answer 1

I would suggest using a mixed model to analyse these data: I assume that you're interested more in the specific ability of each candidate rather than the properties/bias of the reviewers. In this setting, a simple model would be

$$ \mathrm{Grade}_{ij} = \alpha_i + \beta_j + \varepsilon_{ij}, $$

where $i$ ranges over candidates and $j$ over reviewers; $\mathrm{Grade}_{12}$ would be the grade reviewer 2 gives to candidate 1, for example.

This model explains the grade obtained as the sum of:

• a candidate-specific effect $\alpha$, which you can interpret as a measure of their ability,
• a reviewer-specific effect $\beta$, and
• an error term $\varepsilon \sim N(0, \sigma^2)$.

In a mixed model, we assume that these reviewer effects or biases $\beta_j$ are random, distributed independently according to a normal distribution: $\beta_j \sim N(0, \sigma_\beta^2)$.

This model is fair because it tries to separate the candidate ability from the reviewer bias, and it automatically links data across reviewers so that it takes most advantage of the limited sample size.

Assuming your data come in the form (candidate_id, reviewer_id, grade), you can fit the model above using the R commands

library(lme4)
model <- lmer(grade ~ 0 + candidate_id + (1|reviewer_id), data = reviews) # remove intercept

Examining the fixed coefficients (corresponding to candidate_id) would give you a ranking for the candidates, and you can test if they are significantly different from one another using

multcomp::glht(model, linfct = mcp(candidate_id = "Tukey"))

Hope this helps!