I have a set of n texts. Each text is to be assigned a topic by i raters who choose the topic from a set of k discrete topic labels (A, B, C, ... K). So, for each text there are i ratings (categorical data):

Sample data

On this dataset, I want to calculate inter-rater agreement. Given that there are more than two raters, Fleiss' kappa would be a reasonable choice, since Cohen's kappa measures agreement between two raters only.

Given that both the texts to be annotated as well as the raters who annotate the texts are costly, I need to limit the number of texts or the number of raters; otherwise the study would be too expensive.

Now my question: In view of this restriction, which one (number of raters or number of texts) contributes more to the robustness of the inter-rater reliability measure? In other words, is it better to limit the number n of texts to be annotated, or the number i of raters if my goal is to assess how generalisable the obtained results are. The overall goal of the study is to assess how useful the set of k discrete topic labels is for the task of text annotation. Note that the both the raters and the texts in the study are sampled from a potentially very large population of texts/raters. Besides that, the number k of categorical topic labels the raters can choose from is rather high (k equals approximately 200 to 300).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.