1
$\begingroup$

I run competitive events. In our normal event, we have 8 adjudicators split between to categories. Skill and Artistry.

For each category we throw out the high and low scores and average the remaining two scores for the final result. This helps eliminate bias.

I'm trying to find a similar method when there are only 6 judges, three in each category.

I've looked at standard deviation and Quartiles to remove outliers but I'm not satisfied that either is the best choice. My concern is that the Stdev still uses the original mean to determine what is an outlier. Part of me feels that the outlier should be determined based on the "new mean" but then the calculations become so opaque that my team can't follow/duplicate for future events.

With the 1st and 3rd quartile, it seems even more arbitrary to toss the top and bottom 25% but that may be my own cognitive bias.

I'm curious... how would you do this if you were me. Note: In the small groups that I've tested I've found that the results don't vary from taking the simple mean of the three results unless extreme examples are tried... perhaps this isn't worth the effort?

I use googlesheets for the calculations, if that's at all helpful.

Improve this question
$\endgroup$
2
  • $\begingroup$ so you have 3 values from which you want to remove outliers? $\endgroup$
    – rep_ho
    Commented Apr 6, 2023 at 10:49
  • $\begingroup$ Basically. Because it is "judging" we want to eliminate judging bias. $\endgroup$
    – Omar Paloma
    Commented Apr 7, 2023 at 23:29

1 Answer 1

Reset to default
0
$\begingroup$

So it looks like you basically want a summary statistic for three values not influenced by outliers. I suggest using the median value of the three. Not only is this simple and easy to understand, also you can probably hardly do better here, as you don't have enough data to make any more sophisticated assessment of what's an outlier. Also it's as much in line with the procedure for four values as you can get.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I'm not finding that using the mean changes the results except in extreme differences but, as noted above, we want to eliminate bias because these are judges' scores. Example: Two judges' scores are 7 and 7.5, one judge scores a 4. If the judge scoring the 4 is "consistently lower" then the mean will suffice. But if there is evidence of bias, intentional or not, we want to eliminate that score, or at least discourage judges making biased decisions. What's beautiful about the system when there is 4 judges is that it acts as a deterrent in that most judges DO want their score to be counted. $\endgroup$
    – Omar Paloma
    Commented Apr 7, 2023 at 23:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.