# Inter-rater reliability measure with multiple categories per item [duplicate]

I have multiple raters providing categorical ratings. What makes this slightly different from other posted questions is that each rater can provide three, unique categorical ratings. Each rater had a set of survey questions and it was of the form "pick your top 3 choices" from a list. So, multiple raters with three ratings each for each question on my survey.

I've only ever dealt with inter rater reliability with some number of raters choosing a single rating. What would be the best measure of inter rater reliability for this? Thanks for any help.

$$\kappa_0 = \frac{\bar{P} - P_e}{1 - P_e} + \frac{1 - \bar{P}}{Nm_0(1 - P_e)}$$ where $\bar{P}$ is the average proportion of concordant pairs out of all possible pairs of observations for each subject, $P_e=\sum_j p_j^2$ and $p_j$ is the overall proportion of observations in which response category $j$ was selected, $m_0$ is the number of observations per subject, and $N$ is the number of subjects. It can also be shown that, when only one category is selected, $\kappa_0$ asymptotically approaches Cohen's and Fleiss' kappa coefficients.