Accounting for differences between judge scoring

Question

Consider a competition with 10,000 entrants and 200 judges. Each entrant gets scored on a scale of 0-100 by 2 different judges for a total of 20,000 scores.

I want to remove any judge-to-judge variations in their means and standard deviations. To do this I'm using a Z-score for each judge's scores and converting that to a T-score to put in back on a 0-100 scale.

In R I'm doing

df$z_score <- ave(df$score, df$judge, FUN=scale)

df$t_score <- ave(df$score, df$judge, FUN = function(x) rescale(x, mean=50, sd=10, df=FALSE))

in Python the code would be

df['Z-Score'] = df.groupby('judge')['score'].transform(lambda x: stats.zscore(x, ddof=1))

df['T-Score'] = df['Z-Score'].transform(lambda x: x * 10 + 50)

However, for a variety of reasons, some judges only scored a handful of entrants. Let's say between 3 and 20.

Is it valid to calculate a Z-score/T-score for those particular judge's scores as the mean and standard deviations may be skewed due to the small sample or should I run a different test?

Answer 1

If your goal is to remove variations due to the subjectivity of the judges, the simplest way to do this is via some kind of regression model (including GLMs, etc.) where your response score is modelled using the participant and the judge as explanatory variables. This will estimate coefficients for each of the judges, and you can then create an adjusted score that removes the estimated effect of the judge. Alternatively, you could do a similar thing using a linear mixed model using lmer with a random effect for the judges.

Here is an example of some R code to fit a simple linear-mixed-model with a random effect for the judges. The model can then be used to create an adjusted score by taking the predicted score of an entrant which removes the random effect of the judge. Adjusted scores can then be aggregated by participant (averaging their two adjusted scores) to get an overall score for each participant.

library(dplyr);
library(stats);
library(lme4);

#Assume we have a data-frame DATA with variables Participant, Judge, Score 
#    Participant has values 1:10000
#          Judge has values 1:200
#          Score has values 0:100

#Fit a linear-mixed-model with random effects for the judges
#Use this to obtain adjusted scores corresponding to each actual score
SCORE_MODEL         <- lmer(Score ~ factor(Participant) + (1|Judge), data = DATA);
DATA$Score_Adjusted <- predict(SCORE_MODEL, data = DATA);

#Rank the participants by their average adjusted scores
RANKED_DATA <- DATA %>% group_by(Participant) %>% 
                        summarise(Avg_Score_Adjusted = mean(Score_Adjusted)) %>% 
                        ungroup() %>% 
                        arrange(desc(Avg_Score_Adjusted)) %>% 
                        as.data.frame();

This analysis will give you a data-frame containing a ranked list of participants with their average adjusted-scores (after removing the estimated effect of the judge). This can be used as a basis to estimate the "true" ability of each participant. From your question it appears that you are looking for point estimates for a "true score" for each of the participants, but if you need to augment this with interval estimates, you can obtain these using the standard-error estimates for the participant effects in your model.

Answer 2

Look into multilevel models or mixed models, as implemented in R packages such as lme4. The idea is to estimate a varying intercept for each judge. This accounts for the unmeasured heterogeneity among judges, and prevents judges with more trials judged from biasing your parameter estimates. If you use a multilevel model, the data is then independent and identically distributed conditional on the parameters in the model.