# Inter-rater reliability for multiple raters for one item (subject)

I have data with 3 raters, each of whom rated 3 items as follows:

item a item b item c
rater 1 a1 b1 c1
rater 2 a2 b2 c2
rater 3 a3 b3 c3

I successfully calculated the inter-rater reliability for all three items using Krippendorff's alpha library as follows:

krippendorff([[a1, b1, c1], [a2, b2, c2], [a3, b3, c3]])


Now, I am trying to calculate inter-rater reliability per item. For example, for item a, I would like to calculate something like the following:

krippendorff([[a1], [a2], [a3]])


However, both Krippendorff's alpha and Fleiss' kappa always return 0 when the number of items equals 1. Is there an alternative metric that I can use when there is only 1 item?

Thanks in advance.

## 1 Answer

Krippendorff's alpha, like the other distribution-based indexes of chance-adjusted agreement (e.g., kappa and pi), estimates chance agreement using the observed category distributions. With only a single item, you can't estimate chance agreement this way. So you are better off just using percent agreement without adjusting for chance agreement (or gathering more data).