How to calculate inter-rater reliability with multiple raters and multiple categories per item? I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for.
My data set has $r$ raters, $n$ subjects, and $q$ categories. Raters can give each subject anywhere from $0$ to $q$ of the categories.
My understanding is that this means that Fleiss' kappa and Krippendorff's alpha cannot be applied here because they assume that raters give each subject just 1 category. Does anyone have a suggestion for an algorithm that I can use for this data?
 A: You can try the approach of Kraemer (1980) to reframe the problem from one of classification to one of rank ordering. Thus, all selected categories are tied for first place and all non-selected categories are tied for second place. Chance-adjusted agreement can then be calculated using rank correlation coefficients or analysis of variance of the ranks. Naturally, this approach also allows multiple categories to be ranked by raters.
$$
\kappa_0 = \frac{\bar{P} - P_e}{1 - P_e} + \frac{1 - \bar{P}}{Nm_0(1 - P_e)}
$$
where $\bar{P}$ is the average proportion of concordant pairs out of all possible pairs of observations for each subject, $P_e=\sum_j p_j^2$ and $p_j$ is the overall proportion of observations in which response category $j$ was selected, $m_0$ is the number of observations per subject, and $N$ is the number of subjects. It can also be shown that, when only one category is selected, $\kappa_0$ asymptotically approaches Cohen's and Fleiss' kappa coefficients.
References
Kraemer, H. C. (1980). Extension of the kappa coefficient. Biometrics, 36(2), 207–16.
