I have data with 3 raters, each of whom rated 3 items as follows:

item a item b item c
rater 1 a1 b1 c1
rater 2 a2 b2 c2
rater 3 a3 b3 c3

I successfully calculated the inter-rater reliability for all three items using Krippendorff's alpha library as follows:

krippendorff([[a1, b1, c1], [a2, b2, c2], [a3, b3, c3]])

Now, I am trying to calculate inter-rater reliability per item. For example, for item a, I would like to calculate something like the following:

krippendorff([[a1], [a2], [a3]])

However, both Krippendorff's alpha and Fleiss' kappa always return 0 when the number of items equals 1. Is there an alternative metric that I can use when there is only 1 item?

Thanks in advance.


Krippendorff's alpha, like the other distribution-based indexes of chance-adjusted agreement (e.g., kappa and pi), estimates chance agreement using the observed category distributions. With only a single item, you can't estimate chance agreement this way. So you are better off just using percent agreement without adjusting for chance agreement (or gathering more data).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.