# Calculating interrater agreement for multiple choice questions [duplicate]

I'm trying to figure out how to calculate interrater agreement so I can aggregate raters' responses.

I'm asking 5 raters to select topics (out of 6 topics: A, B, C, D, E or F) that describe texts they read (1 = topic relevant to text, 0 = topic irrelevant to text). They can choose as many topics as they like (these are multiple choice questions).

For example, these are the ratings for the first text:

#          A  B  C  D  E  F
# Rater_1  0  0  0  1  0  0
# Rater_2  1  0  0  1  0  0
# Rater_3  1  0  0  1  0  0
# Rater_4  1  0  0  0  0  0
# Rater_5  0  0  0  1  0  0

# Results for text 1:
# 3 raters selected topic A
# 4 raters selected topic D


And another example (second text):

#          A  B  C  D  E  F
# Rater_1  1  1  0  1  0  0
# Rater_2  1  1  0  1  0  0
# Rater_3  1  1  0  0  0  0
# Rater_4  0  1  0  0  0  0
# Rater_5  1  0  0  1  0  0

# Results for text 2:
# 4 raters selected topic A
# 4 raters selected topic B
# 3 raters selected topic D


What would be the right way to aggregate raters' responses for each text:

1) Should I select the topic with the most votes (e.g., for text 1, topic D; for text 2, topics A+B)?

2) Should I select any topic the majority selected, i.e., more than 3 raters said topic is relevant (e.g., for text 1, topics A+D; for text 2, topics A+B+D)?

3) Should I do it any other way?

Thanks!

Edit: Using R.

• Half-answering my own question: Fleiss' Kappa seems to be relevant for this, since it measures IRA in categorical questions rated by multiple judges, however the fact that the categories in my questions are not mutually exclusive (you can choose several topics for each question/text) seems to be a problem, as I read that Fleiss' Kappa needs mutually exclusive categories. Does anyone have any idea what I should do? – David Spivak Feb 16 '16 at 10:27

$$\kappa_0 = \frac{\bar{P} - P_e}{1 - P_e} + \frac{1 - \bar{P}}{Nm_0(1 - P_e)}$$ where $\bar{P}$ is the average proportion of concordant pairs out of all possible pairs of observations for each subject, $P_e=\sum_j p_j^2$ and $p_j$ is the overall proportion of observations in which response category $j$ was selected, $m_0$ is the number of observations per subject, and $N$ is the number of subjects. It can also be shown that, when only one category is selected, $\kappa_0$ asymptotically approaches Cohen's and Fleiss' kappa coefficients.