I have a dataset of 50 annotators that highlight(codified as 1) or not (codified as 0) a text, according to 2 different rules (call them A and B). The dataset is divided into a column for each word of the text. Here a simplified example of the database:

Annotator Rule "The" "cat" "is" "on" "the" "table"
Frank A 0 0 1 1 1 0
Bill B 1 0 1 0 1 0
Jay A 0 0 1 1 1 0
Ellen B 1 0 0 1 1 0

I would like to compare the agreement between the raters in A and raters in B for each word and see if there is an overlapping in what the annotators highlighted, even if the instructions were different. I am quite new to using R and I am looking for the best way to proceed in these kinds of analysis. I have considered Krippendorff's alpha, but it supports only one variable at a time. Furthermore, I want to have info for each word of the text, and not a general value. Thank you in advance for your help.

New contributor
Kashelhirst is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

Kashelhirst is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.