1
$\begingroup$

I am working on a dataset that has three raters. Rating are Yes/No. I have a set of ratings where all raters said No. I used a R package and excel formulas to calculate the kappa score. Both return NaN. I get why it returns NaN. Because the expected probability is 1, the denominator is 0. Hence the NaN. It seems the case is valid when all raters mention Yes to all instances.

Isn't this a perfect agreement?

Here is the data to reproduce

#    Rater 1    Rater 2    Rater 3    Yes    No
1        Yes        Yes        Yes      3     0
2        Yes        Yes        Yes      3     0
3        Yes        Yes        Yes      3     0
4        Yes        Yes        Yes      3     0
5        Yes        Yes        Yes      3     0
6        Yes        Yes        Yes      3     0

Pe = (sum(Yes)/N*n)^2 + (sum(No)/N*n)^2
   = 18/6*3 + 0/6*3
   = 1

Po = 1/(N*n*(n-10)*(Sum_of_squares(all_ratings) - N*n)
   = 1/(6*3*2) * ((3^2+3^2+...+0^2+0^2)-6*3)
   = 1/36 * (54-18)
   = 1/36 * 36
   = 1

kappa = Po - Pe / 1 - Pe
      = 1 - 1 / 1 - 1
      = NaN
$\endgroup$

1 Answer 1

0
$\begingroup$

Your by-chance error rate is zero, since all of the raters give the same “yes” every time. Your kappa calculation, in some sense, doesn’t even know that “no” is an option for the raters. When there is just one option, it doesn’t really make sense to calculate an agreement statistic: perfect agreement is inevitable.

You might be interested in something along the lines of Laplace’s rule of succession to handle the fact that there are only “yes” responses in the data, yet you know “no” to be an option.

Note that this is not an issue if the raters give both responses yet agree every time, something like each rater saying YNYYNN

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.