I have data with 3 raters, each of whom rated 3 items as follows:
|item a||item b||item c|
I successfully calculated the inter-rater reliability for all three items using Krippendorff's alpha library as follows:
krippendorff([[a1, b1, c1], [a2, b2, c2], [a3, b3, c3]])
Now, I am trying to calculate inter-rater reliability per item. For example, for item a, I would like to calculate something like the following:
krippendorff([[a1], [a2], [a3]])
However, both Krippendorff's alpha and Fleiss' kappa always return 0 when the number of items equals 1. Is there an alternative metric that I can use when there is only 1 item?
Thanks in advance.