# How to calculate inter-rater reliability with multiple raters and multiple categories per item? [duplicate]

I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for.

My data set has $r$ raters, $n$ subjects, and $q$ categories. Raters can give each subject anywhere from $0$ to $q$ of the categories.

My understanding is that this means that Fleiss' kappa and Krippendorff's alpha cannot be applied here because they assume that raters give each subject just 1 category. Does anyone have a suggestion for an algorithm that I can use for this data?

• Small note: you use $k$ and $n$ to denote two different things in the title and the body of the post. – Sycorax says Reinstate Monica Apr 5 '15 at 2:26

$$\kappa_0 = \frac{\bar{P} - P_e}{1 - P_e} + \frac{1 - \bar{P}}{Nm_0(1 - P_e)}$$ where $\bar{P}$ is the average proportion of concordant pairs out of all possible pairs of observations for each subject, $P_e=\sum_j p_j^2$ and $p_j$ is the overall proportion of observations in which response category $j$ was selected, $m_0$ is the number of observations per subject, and $N$ is the number of subjects. It can also be shown that, when only one category is selected, $\kappa_0$ asymptotically approaches Cohen's and Fleiss' kappa coefficients.