# How to calculate inter-rater reliability with multiple raters and multiple categories per item? [duplicate]

I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for.

My data set has $r$ raters, $n$ subjects, and $q$ categories. Raters can give each subject anywhere from $0$ to $q$ of the categories.

My understanding is that this means that Fleiss' kappa and Krippendorff's alpha cannot be applied here because they assume that raters give each subject just 1 category. Does anyone have a suggestion for an algorithm that I can use for this data?

## marked as duplicate by Scortchi♦Apr 21 '16 at 10:16

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

• Small note: you use $k$ and $n$ to denote two different things in the title and the body of the post. – Sycorax Apr 5 '15 at 2:26

## 1 Answer

You can try the approach of Kraemer (1980) to reframe the problem from one of classification to one of rank ordering. Thus, all selected categories are tied for first place and all non-selected categories are tied for second place. Chance-adjusted agreement can then be calculated using rank correlation coefficients or analysis of variance of the ranks. Naturally, this approach also allows multiple categories to be ranked by raters.

$$\kappa_0 = \frac{\bar{P} - P_e}{1 - P_e} + \frac{1 - \bar{P}}{Nm_0(1 - P_e)}$$ where $\bar{P}$ is the average proportion of concordant pairs out of all possible pairs of observations for each subject, $P_e=\sum_j p_j^2$ and $p_j$ is the overall proportion of observations in which response category $j$ was selected, $m_0$ is the number of observations per subject, and $N$ is the number of subjects. It can also be shown that, when only one category is selected, $\kappa_0$ asymptotically approaches Cohen's and Fleiss' kappa coefficients.

References

Kraemer, H. C. (1980). Extension of the kappa coefficient. Biometrics, 36(2), 207–16.