Consider a competition with 10,000 entrants and 200 judges. Each entrant gets scored on a scale of 0-100 by 2 different judges for a total of 20,000 scores.

I want to remove any judge-to-judge variations in their means and standard deviations. To do this I'm using a Z-score for each judge's scores and converting that to a T-score to put in back on a 0-100 scale.

In R I'm doing

df$z_score <- ave(df$score, df$judge, FUN=scale)

df$t_score <- ave(df$score, df$judge, FUN = function(x) rescale(x, mean=50, sd=10, df=FALSE))

in Python the code would be

df['Z-Score'] = df.groupby('judge')['score'].transform(lambda x: stats.zscore(x, ddof=1))

df['T-Score'] = df['Z-Score'].transform(lambda x: x * 10 + 50)

However, for a variety of reasons, some judges only scored a handful of entrants. Let's say between 3 and 20.

Is it valid to calculate a Z-score/T-score for those particular judge's scores as the mean and standard deviations may be skewed due to the small sample or should I run a different test?


2 Answers 2


If your goal is to remove variations due to the subjectivity of the judges, the simplest way to do this is via some kind of regression model (including GLMs, etc.) where your response score is modelled using the participant and the judge as explanatory variables. This will estimate coefficients for each of the judges, and you can then create an adjusted score that removes the estimated effect of the judge. Alternatively, you could do a similar thing using a linear mixed model using lmer with a random effect for the judges.

Here is an example of some R code to fit a simple linear-mixed-model with a random effect for the judges. The model can then be used to create an adjusted score by taking the predicted score of an entrant which removes the random effect of the judge. Adjusted scores can then be aggregated by participant (averaging their two adjusted scores) to get an overall score for each participant.


#Assume we have a data-frame DATA with variables Participant, Judge, Score 
#    Participant has values 1:10000
#          Judge has values 1:200
#          Score has values 0:100

#Fit a linear-mixed-model with random effects for the judges
#Use this to obtain adjusted scores corresponding to each actual score
SCORE_MODEL         <- lmer(Score ~ factor(Participant) + (1|Judge), data = DATA);
DATA$Score_Adjusted <- predict(SCORE_MODEL, data = DATA);

#Rank the participants by their average adjusted scores
RANKED_DATA <- DATA %>% group_by(Participant) %>% 
                        summarise(Avg_Score_Adjusted = mean(Score_Adjusted)) %>% 
                        ungroup() %>% 
                        arrange(desc(Avg_Score_Adjusted)) %>% 

This analysis will give you a data-frame containing a ranked list of participants with their average adjusted-scores (after removing the estimated effect of the judge). This can be used as a basis to estimate the "true" ability of each participant. From your question it appears that you are looking for point estimates for a "true score" for each of the participants, but if you need to augment this with interval estimates, you can obtain these using the standard-error estimates for the participant effects in your model.

  • $\begingroup$ Should lmer take a very long time to run? $\endgroup$
    – agf1997
    Jun 7, 2018 at 0:00
  • $\begingroup$ I would think lmer would have little difficulty with a data frame with 20,000 observations. I have run it on similar sized data sets before with no problems. $\endgroup$
    – Ben
    Jun 7, 2018 at 0:04
  • $\begingroup$ 14204 obs. and 7 variables I run that line and R just hangs $\endgroup$
    – agf1997
    Jun 7, 2018 at 0:04
  • $\begingroup$ This hits the limits of my knowledge - you might be able to get some advice posting a question about it on Stackoverflow.SE. Those guys know a lot more about the computational aspects of R. $\endgroup$
    – Ben
    Jun 7, 2018 at 0:10
  • $\begingroup$ Thanks ... that's understandable. It's still just sitting there. $\endgroup$
    – agf1997
    Jun 7, 2018 at 0:41

Look into multilevel models or mixed models, as implemented in R packages such as lme4. The idea is to estimate a varying intercept for each judge. This accounts for the unmeasured heterogeneity among judges, and prevents judges with more trials judged from biasing your parameter estimates. If you use a multilevel model, the data is then independent and identically distributed conditional on the parameters in the model.

  • $\begingroup$ I’m not at all familiar with multilevel or mixed models. Perhaps I just need to do more research but they appear to be regression techniques. How does that help me create some form of unbiased score? $\endgroup$
    – agf1997
    May 25, 2018 at 17:46
  • $\begingroup$ I'm not sure how these techniques apply. I've been looking into them and from what I can read they deal with situations where there are hierarchies. What is the hierarchy in this case? An example would be helpful. $\endgroup$
    – agf1997
    May 30, 2018 at 17:24
  • $\begingroup$ The hierarchy is that the judging of entrants are nested within judges. $\endgroup$ May 30, 2018 at 17:32
  • $\begingroup$ How come "entrants are nested within judges"? Each judge rated various entrants (from 3 to 20). $\endgroup$
    – ttnphns
    May 30, 2018 at 18:55
  • $\begingroup$ Entrants are nested within judges means exactly that: each judge rated various entrants. So there is an entrant effect on their rating, and a judge effect. A hierarchical model allows you to partially pool entrants within judges. $\endgroup$ May 30, 2018 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.