I want to calculate the Inter-rater Reliability of an annotated dataset, where the number of raters for each subject is variable (3-5 raters) and subjects can be annotated with one of 7 categories.
Cohen's kappa is not applicable since it requires exactly 2 raters. Also, Fleiss's kappa is not applicable as it requires the exact number of raters for each subject. I've read about the generalized kappa by Uebersax (1982) and it seems to be the most suitable solution. However, I didn't find any implementation or tool for calculating the generalized kappa.