2
$\begingroup$

I have been tasked to approximate the error rate of a research procedure.

The only data I have available are reports which specify the number of targets (n), the number of raters (k) and give pairwise inter-rater reliability (Fleiss' kappa) values. The 'true values' of the targets are unknown, as are the values that raters assigned to individual targets.

Is there a way to estimate how many errors each single rater might have made based on that information?

I came across these posts:

Standard Error of Measurement (SEM) in Inter-rater reliability

How to compute the standard error of measurement (SEM) from a reliability estimate?

But I am unsure whether they address the issue I am facing. Any hints appreciated!

$\endgroup$

2 Answers 2

1
$\begingroup$

I don't think you can.

First, you don't know the truth. Maybe every judge makes the same errors.

Second, kappa is chance-corrected agreement. So you need to know the base rate to interpret it:

Example 1:

Two judges, A and B.

  • Both agree on Yes: 82
  • Both agree on No: 1
  • A Yes, B No: 9
  • A No, B Yes: 9

Judges agree 83% of the time. Let's say when both agree, they're both right, when they disagree, 50% are right. Each judge is correct 92% of the time. Kappa is 0.

  • Both agree on Yes: 40
  • Both agree on No: 40
  • A Yes, B No: 10
  • A No, B Yes: 10

80% agreement, 90% correct, kappa 0.6.

$\endgroup$
0
$\begingroup$

You can't. Whether two raters agree with each other in general has nothing at all to do with whether they agree with the truth. You could see two raters agree 100% of the time and find that they are both always wrong. You could see two raters agree 0% of the time and find that is always right and one is always wrong. For multinomial problems, you could even see two raters never agree, and have them both be wrong 100% of the time in different ways.

You might be able to get somewhere by inferring that if multiple raters call the same class that it's probably correct, but without any information on how accurate the individual raters are or how they relate to each other, we have no way of relating the raters' calls to the ground truth.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.