I am working on machine translation evaluation, and am looking at ratings given by humans who are judging the quality of sentences produced by machine translation systems. The evaluators give fluency and adequacy (how well the information is preserved) scores to sentences. The data is from NTCIR7 for those that are interested.
For the test data I have, there were 3 human evaluators, and they rated 100 sentences, from 5 different systems, so 500 sentences in total. The scores they gave were between 1 and 5 (which I normalised to 0.0-1.0).
Problem: It seems that one human evaluator A is biased towards giving much more generous fluency scores. You can see this in the rough chart below, that A gives many more sentences a perfect score of 1 when compared to B and C.
- What statistical methods can I use to try to remove this bias? Is bias even the right word?
- With only 3 evaluators, is it a good idea to modify the scores at all? Does that count as 'cheating' to get the answer I think is right?
Graphs: (sorry, the highest score is on the left)
Fluency: A gives many more perfect 1 scores than B or C
Adequacy: C gives more low 0.25 scores, but fewer 0 scores. I am less worried about this as it seems to average out. But it is still kind of troubling.