Consider a competition with 10,000 entrants and 200 judges. Each entrant gets scored on a scale of 0-100 by 2 different judges for a total of 20,000 scores.

I want to remove any judge-to-judge variations in their means and standard deviations. To do this I'm using a Z-score for each judge's scores and converting that to a T-score to put in back on a 0-100 scale.

In R I'm doing

df$z_score <- ave(df$score, df$judge, FUN=scale)

df$t_score <- ave(df$score, df$judge, FUN = function(x) rescale(x, mean=50, sd=10, df=FALSE))

in Python the code would be

df['Z-Score'] = df.groupby('judge')['score'].transform(lambda x: stats.zscore(x, ddof=1))

df['T-Score'] = df['Z-Score'].transform(lambda x: x * 10 + 50)

However, for a variety of reasons, some judges only scored a handful of entrants. Let's say between 3 and 20.

Is it valid to calculate a Z-score/T-score for those particular judge's scores as the mean and standard deviations may be skewed due to the small sample or should I run a different test?


If your goal is to remove variations due to the subjectivity of the judges, the simplest way to do this is via some kind of regression model (including GLMs, etc.) where your response score is modelled using the participant and the judge as explanatory variables. This will estimate coefficients for each of the judges, and you can then create an adjusted score that removes the estimated effect of the judge. Alternatively, you could do a similar thing using a linear mixed model using lmer with a random effect for the judges.

Here is an example of some R code to fit a simple linear-mixed-model with a random effect for the judges. The model can then be used to create an adjusted score by taking the predicted score of an entrant which removes the random effect of the judge. Adjusted scores can then be aggregated by participant (averaging their two adjusted scores) to get an overall score for each participant.


#Assume we have a data-frame DATA with variables Participant, Judge, Score 
#    Participant has values 1:10000
#          Judge has values 1:200
#          Score has values 0:100

#Fit a linear-mixed-model with random effects for the judges
#Use this to obtain adjusted scores corresponding to each actual score
SCORE_MODEL         <- lmer(Score ~ factor(Participant) + (1|Judge), data = DATA);
DATA$Score_Adjusted <- predict(SCORE_MODEL, data = DATA);

#Rank the participants by their average adjusted scores
RANKED_DATA <- DATA %>% group_by(Participant) %>% 
                        summarise(Avg_Score_Adjusted = mean(Score_Adjusted)) %>% 
                        ungroup() %>% 
                        arrange(desc(Avg_Score_Adjusted)) %>% 

This analysis will give you a data-frame containing a ranked list of participants with their average adjusted-scores (after removing the estimated effect of the judge). This can be used as a basis to estimate the "true" ability of each participant. From your question it appears that you are looking for point estimates for a "true score" for each of the participants, but if you need to augment this with interval estimates, you can obtain these using the standard-error estimates for the participant effects in your model.

| cite | improve this answer | |
  • $\begingroup$ Should lmer take a very long time to run? $\endgroup$ – agf1997 Jun 7 '18 at 0:00
  • $\begingroup$ I would think lmer would have little difficulty with a data frame with 20,000 observations. I have run it on similar sized data sets before with no problems. $\endgroup$ – Ben - Reinstate Monica Jun 7 '18 at 0:04
  • $\begingroup$ 14204 obs. and 7 variables I run that line and R just hangs $\endgroup$ – agf1997 Jun 7 '18 at 0:04
  • $\begingroup$ This hits the limits of my knowledge - you might be able to get some advice posting a question about it on Stackoverflow.SE. Those guys know a lot more about the computational aspects of R. $\endgroup$ – Ben - Reinstate Monica Jun 7 '18 at 0:10
  • $\begingroup$ Thanks ... that's understandable. It's still just sitting there. $\endgroup$ – agf1997 Jun 7 '18 at 0:41

Look into multilevel models or mixed models, as implemented in R packages such as lme4. The idea is to estimate a varying intercept for each judge. This accounts for the unmeasured heterogeneity among judges, and prevents judges with more trials judged from biasing your parameter estimates. If you use a multilevel model, the data is then independent and identically distributed conditional on the parameters in the model.

| cite | improve this answer | |
  • $\begingroup$ I’m not at all familiar with multilevel or mixed models. Perhaps I just need to do more research but they appear to be regression techniques. How does that help me create some form of unbiased score? $\endgroup$ – agf1997 May 25 '18 at 17:46
  • $\begingroup$ I'm not sure how these techniques apply. I've been looking into them and from what I can read they deal with situations where there are hierarchies. What is the hierarchy in this case? An example would be helpful. $\endgroup$ – agf1997 May 30 '18 at 17:24
  • $\begingroup$ The hierarchy is that the judging of entrants are nested within judges. $\endgroup$ – Brash Equilibrium May 30 '18 at 17:32
  • $\begingroup$ How come "entrants are nested within judges"? Each judge rated various entrants (from 3 to 20). $\endgroup$ – ttnphns May 30 '18 at 18:55
  • $\begingroup$ Entrants are nested within judges means exactly that: each judge rated various entrants. So there is an entrant effect on their rating, and a judge effect. A hierarchical model allows you to partially pool entrants within judges. $\endgroup$ – Brash Equilibrium May 30 '18 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.