I have multiple raters providing categorical ratings. What makes this slightly different from other posted questions is that each rater can provide three, unique categorical ratings. Each rater had a set of survey questions and it was of the form "pick your top 3 choices" from a list. So, multiple raters with three ratings each for each question on my survey.

I've only ever dealt with inter rater reliability with some number of raters choosing a single rating. What would be the best measure of inter rater reliability for this? Thanks for any help.


Kramer (1980) proposed a method for assessing inter-rater reliability for tasks in which raters could select multiple categories for each object of measurement. The intuition behind this method is to reframe the problem from one of classification to one of rank ordering. Thus, all selected categories are tied for first place and all non-selected categories are tied for second place. Chance-adjusted agreement can then be calculated using rank correlation coefficients or analysis of variance of the ranks. Naturally, this approach also allows multiple categories to be ranked by raters.

$$ \kappa_0 = \frac{\bar{P} - P_e}{1 - P_e} + \frac{1 - \bar{P}}{Nm_0(1 - P_e)} $$ where $\bar{P}$ is the average proportion of concordant pairs out of all possible pairs of observations for each subject, $P_e=\sum_j p_j^2$ and $p_j$ is the overall proportion of observations in which response category $j$ was selected, $m_0$ is the number of observations per subject, and $N$ is the number of subjects. It can also be shown that, when only one category is selected, $\kappa_0$ asymptotically approaches Cohen's and Fleiss' kappa coefficients.

A clever solution, but not one that I've ever seen used in an article.


Kraemer, H. C. (1980). Extension of the kappa coefficient. Biometrics, 36(2), 207–16.


Not the answer you're looking for? Browse other questions tagged or ask your own question.