# Calculating interrater agreement for multiple choice questions [duplicate]

I'm trying to figure out how to calculate interrater agreement so I can aggregate raters' responses.

I'm asking 5 raters to select topics (out of 6 topics: A, B, C, D, E or F) that describe texts they read (1 = topic relevant to text, 0 = topic irrelevant to text). They can choose as many topics as they like (these are multiple choice questions).

For example, these are the ratings for the first text:

#          A  B  C  D  E  F
# Rater_1  0  0  0  1  0  0
# Rater_2  1  0  0  1  0  0
# Rater_3  1  0  0  1  0  0
# Rater_4  1  0  0  0  0  0
# Rater_5  0  0  0  1  0  0

# Results for text 1:
# 3 raters selected topic A
# 4 raters selected topic D


And another example (second text):

#          A  B  C  D  E  F
# Rater_1  1  1  0  1  0  0
# Rater_2  1  1  0  1  0  0
# Rater_3  1  1  0  0  0  0
# Rater_4  0  1  0  0  0  0
# Rater_5  1  0  0  1  0  0

# Results for text 2:
# 4 raters selected topic A
# 4 raters selected topic B
# 3 raters selected topic D


What would be the right way to aggregate raters' responses for each text:

1) Should I select the topic with the most votes (e.g., for text 1, topic D; for text 2, topics A+B)?

2) Should I select any topic the majority selected, i.e., more than 3 raters said topic is relevant (e.g., for text 1, topics A+D; for text 2, topics A+B+D)?

3) Should I do it any other way?

Thanks!

Edit: Using R.

• Half-answering my own question: Fleiss' Kappa seems to be relevant for this, since it measures IRA in categorical questions rated by multiple judges, however the fact that the categories in my questions are not mutually exclusive (you can choose several topics for each question/text) seems to be a problem, as I read that Fleiss' Kappa needs mutually exclusive categories. Does anyone have any idea what I should do? – David Spivak Feb 16 '16 at 10:27

Kramer (1980) proposed a method for assessing inter-rater reliability for tasks in which raters could select multiple categories for each object of measurement. The intuition behind this method is to reframe the problem from one of classification to one of rank ordering. Thus, all selected categories are tied for first place and all non-selected categories are tied for second place. Chance-adjusted agreement can then be calculated using rank correlation coefficients or analysis of variance of the ranks. Naturally, this approach also allows multiple categories to be ranked by raters.

$$\kappa_0 = \frac{\bar{P} - P_e}{1 - P_e} + \frac{1 - \bar{P}}{Nm_0(1 - P_e)}$$ where $\bar{P}$ is the average proportion of concordant pairs out of all possible pairs of observations for each subject, $P_e=\sum_j p_j^2$ and $p_j$ is the overall proportion of observations in which response category $j$ was selected, $m_0$ is the number of observations per subject, and $N$ is the number of subjects. It can also be shown that, when only one category is selected, $\kappa_0$ asymptotically approaches Cohen's and Fleiss' kappa coefficients.

A clever solution, but not one that I've ever seen used in an article.

References

Kraemer, H. C. (1980). Extension of the kappa coefficient. Biometrics, 36(2), 207–16.