Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr package and the kappa fleiss function.

I would expect the kappa to be also equals to 1 when the table is full of 0, as the 0 would represent agreement on 'No' and the full of 1s would represent agreement of 'Yes'.

What is the right way to interpret the results?

share|improve this question

This is an issue with the formulation of many chance-adjusted indices of inter-rater reliability that estimate chance agreement using a distribution-based approach. Basically, you are getting a score of NaN because, with only one observed response category, $\kappa$ estimates the chance agreement $(p_c)$ as $1$. Using the formula below, a $p_c$ of $1$ results in a denominator of $0$, which cannot be computed:

$$ \kappa = \frac{p_a - p_c}{1 - p_c} $$

In this case, there is perfect observed agreement (i.e., $p_a=1$) and I would report that instead of $\kappa$.

If you are interested in these issues and the assumptions made by different reliability indices, see:

Zhao, X., Liu, J. S., & Deng, K. (2012). Assumptions behind inter-coder reliability indices. In C. T. Salmon (Ed.), Communication Yearbook (pp. 418–480). Routledge.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.