1
$\begingroup$

In a test of inter-rater agreement, I planned to report a kappa and lower+upper confidence bounds. However, I found 100% perfect agreement between the two raters. Is it possible to produce a confidence interval from this? In addition, is it appropriate to report a Kappa of 1?

$\endgroup$
0
$\begingroup$

If your point and interval estimates are are $1.00$ then you can and should report that. However, it's important to interpret these results in the context of the design of the reliability experiment (i.e., are these estimates good?). For example, how large was the sample of objects being rated and how large was the sample of raters? Were these samples large and representative enough of the populations of interest to be trustworthy? In my experience, it is pretty rare to see perfect agreement on a large sample, so I'm wondering if this result may be revealing underlying issues in the experimental design.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.