I'm using inter-rater agreement to evaluate the agreement in my rating dataset. I have a set of N examples distributed among M raters. Not all raters voted every item, so I have N x M votes as the upper bound. So let's say the rater i gives the following votes the the N items, for a given N=5 and M=3, where in the array at position j there is the j-th item:

rater[1] = [1,3,0,5,5]
rater[2] = [0,3,1,5,2]
rater[3] = [1,2,0,5,3]

where 0 meaning that the voter did not express any option about item in position j. Now, I cannot use the Cohen's Kappa, since it requires to have almost two rathers, so I think to use the Alpha Krippendorff of NLTK or the multi-kappa.

In my dataset

  • Votes eventually can be sparse, i.e there can be items that have few votes hence like the worst case of

    rater[i] = [0, 0, ...,j, ..., 0]

so the item j could have just one vote by the rater i in the whole dataset.

  • Each item must have at least one vote, hence there are no items with a zero array.
  • The numbers of raters M is less than the numbers of items N, M < N.

Which is the best approach as for the NLTK metrics package implementation?


1 Answer 1


I found this solution and it was useful for my case so you probably check it out. Here's the possible solution for your dataset.

import krippendorff
data = [[1,3,0,5,5],

kappa = krippendorff.alpha(data)

It works with Python 3.4+ and don't forget to install dependencies

pip install numpy krippendorff
  • 1
    $\begingroup$ Thanks a lot Abu for your solution! $\endgroup$ Oct 5, 2019 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.