# How should I interpret Fleiss' kappa when it equals NaN?

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr package and the kappa fleiss function.

I would expect the kappa to be also equals to 1 when the table is full of 0, as the 0 would represent agreement on 'No' and the full of 1s would represent agreement of 'Yes'.

What is the right way to interpret the results?

-

This is an issue with the formulation of many chance-adjusted indices of inter-rater reliability that estimate chance agreement using a distribution-based approach. Basically, you are getting a score of NaN because, with only one observed response category, $\kappa$ estimates the chance agreement $(p_c)$ as $1$. Using the formula below, a $p_c$ of $1$ results in a denominator of $0$, which cannot be computed:
$$\kappa = \frac{p_a - p_c}{1 - p_c}$$
In this case, there is perfect observed agreement (i.e., $p_a=1$) and I would report that instead of $\kappa$.