Assessing rater bias where one rater has given one very high rating and the remainder very low ratings What is a good statistical test to check if there is a bias in judging in a situation that there is one judge that gave extreme scores (high score for one of the contestant and very low scores on the rest of the contestants)?  Here is the actual data in the contest:
                 contestant 1      contestant 2     contestant 3    contestant 4
judge 1            83.03               96.5             88.5           90.5
judge 2            67.15               89.9             85.36          89.85
judge 3            72.05               84.6             78.95          85
judge 4            86.95               93.3             88             94.1
judge 5            44                  65.15            52.45          96.05

Thank you very much!
 A: You could measure agreement in ratings across judges with inter-rater reliability statistics. This would tell you whether the judging of contestants is consistent across judges.
There may be a more sophisticated way of doing this, but I might naively try dropping out each of the five judges individually looking at how the reliability changes.
But with such a small sample, I don't think you'll get particularly strong answers whatever you do.
A: You won't be able to demonstrate bias, but you can try to establish whether the 96.05 is an outlier using Dixon's Test for Outliers.  If these judges went on to judge these same contestants on another task/domain, you could test for the replicability of this unusual result for Judge 5 and Contestant 4.
A: You could think of this as a test of variances. Judge 5's scores will get more weight because the variability of the scores is higher. 
This test would be for the equality of two variances. It's in most intro stat books, and even in Excel, which provides the following results for judge 5 versus judge 1-4
F-Test Two-Sample for Variances     
Variable 1  Variable 2

Mean    64.4125 85.85875
Variance    520.415625  60.13891833
Observations    4   16
df  3   15
F   8.653558119 
P(F<=f) one-tail    0.001424952 
F Critical one-tail 3.287382105 
This does show judge 5 is significantly more variable than the other judges, but frankly I would be careful of a result like this because of the amount of "fishing" involved. You're looking at this after-the-fact, with several possible hypotheses available (just to start, there are equivalent tests of judge 1 against 2,3,4,5, judge 2 against 1,3,4,5, etc.) 
It's also possible that even if you are observing something, it might not be what you think. You might not be observing bias for/against contestants for a consistent tendency to view things in a different way -- sort of like umpires being willing to call a high strike in American baseball, with various pitchers tending to use/not use a high strike.
If you had more contest results, you could compare judge 5 (and others) versus some overall norm. That gets around the fact that with a small sample of judges and ratings (and a posthoc analysis!) you can't really get above the suspicion level.
