I'm using inter-rater agreement to evaluate the agreement in my rating dataset. I have a set of
N examples distributed among
M raters. Not all raters voted every item, so I have
N x M votes as the upper bound.
So let's say the rater
i gives the following votes the the
N items, for a given
M=3, where in the array at position
j there is the
rater = [1,3,0,5,5] rater = [0,3,1,5,2] rater = [1,2,0,5,3]
0 meaning that the voter did not express any option about item in position
Now, I cannot use the Cohen's Kappa, since it requires to have almost two rathers, so I think to use the Alpha Krippendorff of NLTK or the multi-kappa.
In my dataset
Votes eventually can be sparse, i.e there can be items that have few votes hence like the worst case of
rater[i] = [0, 0, ...,j, ..., 0]
so the item
j could have just one vote by the rater
i in the whole dataset.
- Each item must have at least one vote, hence there are no items with a zero array.
- The numbers of raters
Mis less than the numbers of items
M < N.
Which is the best approach as for the NLTK metrics package implementation?