1
$\begingroup$

Consider the scenario where M performances (eg. singing contest) are being judged by N judges. Each judge awards a score S(m,n) to each performance on a scale of one to one-hundred.

The problem occurs when each judge has an individual style of using the scale. For example Judge A might give the worst performances a minimum of 30 and the best a maximum of 90, while Judge B may handout the minimum as 10 and the maximum as 80.

How do we regularize the scores so as the get the overall correct score? Some justification (link to an article or paper) for the answer would be appreciated.

This is similar to https://stats.stackexchange.com/questions/138973/normalizing-interview-scores, but that question has received no responses.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

It seems like your goal is to ensure that the center and spread of the scores match up so that all centers are the same (i.e. judge a average = judge b average) and all spreads are the same. However, you want the judges scores' to retain their original shapes - that is, if one score was very high and all others were very low, you want to keep that "ranking" and make sure that the top score is still far higher than the very low scores.

This seems to be the perfect situation in which you use standardization. For each individual $i$, performance $m$, judge $n$, you should calculate:

$z_{imn} = \frac{S_i(m,n) - \bar{S}(m,n)}{\sigma(S(m,n))}$

where $z_{imn}$ is the standardized score, $S_i(m,n)$ is the score of individual $i$ for performance $m$ by judge $n$, $\bar{S}(m,n)$ is the average score by judge $n$ for performance $m$, and $\sigma(S(m,n))$ is the standard deviation of the scores by judge $n$ for performance $m$. This will standardize the means and standard deviations such that the mean for each judge will be 0 and the standard deviation for each judge will be 1, but the shapes of the distributions of scores will be retained to account for situations like the "one very high, all others very low" situation detailed above. You would then report the $z_{imn}$ as the standardized scores and use these for your analysis.

Improve this answer
$\endgroup$
4
  • $\begingroup$ What I understand from your answer is that you are finding the Z-score for each S(m,n). I have two concerns regarding this. 1. If the data is not normally distributed, this method will not preserve the actual distribution. 2. Calculation of Z-score needs true mean and standard deviation, the formula is different if the sample mean is used. $\endgroup$
    – UditG
    Nov 29, 2015 at 23:08
  • $\begingroup$ 1. This is incorrect. The original shape will always be preserved. You're merely changing the location and scale of the distribution - not the shape. If the data are Normal, then the standardized version will be Normal. If the data are not Normal, then the standardized version will not be Normal. 2. This is true, but you could consider these to be the population of scores and thus the true mean and standard deviation. In any case, treating the sample mean and standard deviation as the population mean/SD will yield the desired effect. $\endgroup$
    – Matt Brems
    Nov 30, 2015 at 1:45
  • $\begingroup$ Aah thanks for clarifying and for taking the time to answer. $\endgroup$
    – UditG
    Nov 30, 2015 at 2:42
  • $\begingroup$ This is also assuming each performance m is unique. If they're all the same, then we can use judge n's mean and SD instead of judge n and performance m's mean and SD. But if they're different (i.e. one performance is short and one is longer), we can calculate a different mean and SD for each combination of judge and performance. $\endgroup$
    – Matt Brems
    Nov 30, 2015 at 3:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.