I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can give each item one of 3 values (-1,0, and 1).
I tried using weighted kappa to measure the agreement between each pair of raters at a time. However, the results I am getting seem a little strange. When only 2 of the 3 values are used by the raters the result is 0 even if they agree on most items. Example:
(-1,-1; -1 ,-1;-1 , -1; -1, -1;-1 , 0; -1, 0)
even though the two raters gave the same rating to 4/6 items the result is 0.
Is cohen kappa the best suited method for my problem? If not, what's a better alternative.