1
$\begingroup$

I want to calculate the Inter-rater Reliability of an annotated dataset, where the number of raters for each subject is variable (3-5 raters) and subjects can be annotated with one of 7 categories.

Cohen's kappa is not applicable since it requires exactly 2 raters. Also, Fleiss's kappa is not applicable as it requires the exact number of raters for each subject. I've read about the generalized kappa by Uebersax (1982) and it seems to be the most suitable solution. However, I didn't find any implementation or tool for calculating the generalized kappa.

$\endgroup$
7
  • 1
    $\begingroup$ Should we assume you’re looking for a recommendation other than “Implement the formula from Uebersax”? $\endgroup$ Jun 2, 2021 at 20:07
  • $\begingroup$ @AryaMcCarthy No. Either other recommendation or implemented formula will be good. I edited the question. $\endgroup$
    – NST
    Jun 3, 2021 at 9:56
  • $\begingroup$ If you're looking for a software implementation, what platform do you want? R? MATLAB? Python? Excel? $\endgroup$ Jun 4, 2021 at 2:11
  • $\begingroup$ Thanks @JeffreyGirard! I prefer Python or Excel. $\endgroup$
    – NST
    Jun 4, 2021 at 13:57
  • $\begingroup$ Check out agreestat.com there are excel and cloud based implementations. $\endgroup$ Jun 4, 2021 at 14:24

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.