Inter-rater reliability for multiple raters for one item (subject)

Question

I have data with 3 raters, each of whom rated 3 items as follows:

	item a	item b	item c
rater 1	a1	b1	c1
rater 2	a2	b2	c2
rater 3	a3	b3	c3

I successfully calculated the inter-rater reliability for all three items using Krippendorff's alpha library as follows:

krippendorff([[a1, b1, c1], [a2, b2, c2], [a3, b3, c3]])

Now, I am trying to calculate inter-rater reliability per item. For example, for item a, I would like to calculate something like the following:

krippendorff([[a1], [a2], [a3]])

However, both Krippendorff's alpha and Fleiss' kappa always return 0 when the number of items equals 1. Is there an alternative metric that I can use when there is only 1 item?

Thanks in advance.

Answer 1

Krippendorff's alpha, like the other distribution-based indexes of chance-adjusted agreement (e.g., kappa and pi), estimates chance agreement using the observed category distributions. With only a single item, you can't estimate chance agreement this way. So you are better off just using percent agreement without adjusting for chance agreement (or gathering more data).