5
$\begingroup$

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr package and the kappa fleiss function.

I would expect the kappa to be also equals to 1 when the table is full of 0, as the 0 would represent agreement on 'No' and the full of 1s would represent agreement of 'Yes'.

What is the right way to interpret the results?

$\endgroup$

1 Answer 1

3
$\begingroup$

This is an issue with the formulation of many chance-adjusted indices of inter-rater reliability that estimate chance agreement using a distribution-based approach. Basically, you are getting a score of NaN because, with only one observed response category, $\kappa$ estimates the chance agreement $(p_c)$ as $1$. Using the formula below, a $p_c$ of $1$ results in a denominator of $0$, which cannot be computed:

$$ \kappa = \frac{p_a - p_c}{1 - p_c} $$

In this case, there is perfect observed agreement (i.e., $p_a=1$) and I would report that instead of $\kappa$.

If you are interested in these issues and the assumptions made by different reliability indices, see:

Zhao, X., Liu, J. S., & Deng, K. (2012). Assumptions behind inter-coder reliability indices. In C. T. Salmon (Ed.), Communication Yearbook (pp. 418–480). Routledge.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.