This question already has an answer here:
I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for.
My data set has $r$ raters, $n$ subjects, and $q$ categories. Raters can give each subject anywhere from $0$ to $q$ of the categories.
My understanding is that this means that Fleiss' kappa and Krippendorff's alpha cannot be applied here because they assume that raters give each subject just 1 category. Does anyone have a suggestion for an algorithm that I can use for this data?