Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr package and the kappa fleiss function.
I would expect the kappa to be also equals to 1 when the table is full of 0, as the 0 would represent agreement on 'No' and the full of 1s would represent agreement of 'Yes'.
What is the right way to interpret the results?
This is an issue with the formulation of many chance-adjusted indices of inter-rater reliability that estimate chance agreement using a distribution-based approach. Basically, you are getting a score of NaN because, with only one observed response category, $\kappa$ estimates the chance agreement $(p_c)$ as $1$. Using the formula below, a $p_c$ of $1$ results in a denominator of $0$, which cannot be computed:
$$ \kappa = \frac{p_a - p_c}{1 - p_c} $$
In this case, there is perfect observed agreement (i.e., $p_a=1$) and I would report that instead of $\kappa$.
If you are interested in these issues and the assumptions made by different reliability indices, see:
Zhao, X., Liu, J. S., & Deng, K. (2012). Assumptions behind inter-coder reliability indices. In C. T. Salmon (Ed.), Communication Yearbook (pp. 418–480). Routledge.
