Removing human evaluator bias

I am working on machine translation evaluation, and am looking at ratings given by humans who are judging the quality of sentences produced by machine translation systems. The evaluators give fluency and adequacy (how well the information is preserved) scores to sentences. The data is from NTCIR7 for those that are interested.

For the test data I have, there were 3 human evaluators, and they rated 100 sentences, from 5 different systems, so 500 sentences in total. The scores they gave were between 1 and 5 (which I normalised to 0.0-1.0).

Problem: It seems that one human evaluator A is biased towards giving much more generous fluency scores. You can see this in the rough chart below, that A gives many more sentences a perfect score of 1 when compared to B and C.

Question:

• What statistical methods can I use to try to remove this bias? Is bias even the right word?
• With only 3 evaluators, is it a good idea to modify the scores at all? Does that count as 'cheating' to get the answer I think is right?

Graphs: (sorry, the highest score is on the left)

Fluency: A gives many more perfect 1 scores than B or C

Adequacy: C gives more low 0.25 scores, but fewer 0 scores. I am less worried about this as it seems to average out. But it is still kind of troubling.

-

I first want to adress something else. Reliability. I often tell my students that reliability is a neccesary but insufficient precondition for validity. In essence, if the results are not reliable, they cannot be valid. If you have bias in your rater evaluations you should first assess the reliability by computing the interater relaibility. If it is low, then your results are not reliable and therefore drawing any conclusions from your data cannot be valid. If, on the other hand inter rater reliability is high then you have a rater bias such as leniency or severity. In this case there are some things you can do. For example, standardizing the rater evaluations by computing the mean differentiation score.

The one thing I would definetly NOT DO is average the ratings across the three raters until reliability is established. A quick story. Testing a claim by a mouthwash company that used two raters to judge the breath of people who consumed pizza a garlic for several hours after using one of several brands of mouthwash showed an significant ANOVA main effect that favored the mouthwash company when the judges ratings were average. Inter rater reliabiliy was about .3 (r). Therefore, the main effect cannot be valid because the judges never agreed with one another - in other words, their judgements were unreliable and as I said at the top of this post, if it is not reliable it cannot be valid.

Dr. Doug

-