I have a set of *n* texts. Each text is to be assigned a topic by *i* raters who choose the topic from a set of *k* discrete topic labels (A, B, C, ... K). So, for each text there are *i* ratings (categorical data):

[![Sample data][1]][1]


  [1]: https://i.sstatic.net/cx1T6.png

On this dataset, I want to calculate inter-rater agreement. Given that there are more than two raters, Fleiss' kappa would be a reasonable choice, since Cohen's kappa measures agreement between two raters only.

Given that both the texts to be annotated as well as the raters who annotate the texts are costly, I need to limit the number of texts **or** the number of raters; otherwise the study would be too expensive.

Now my question: In view of this restriction, which one (number of raters or number of texts) contributes more to the robustness of the inter-rater reliability measure? In other words, is it better to limit the number *n* of texts to be annotated, or the number *i* of raters if my goal is to assess how generalisable the obtained results are. The overall goal of the study is to assess how useful the set of *k* discrete topic labels is for the task of text annotation. Note that the both the raters and the texts in the study are sampled from a potentially very large population of texts/raters. Besides that, the number *k* of categorical topic labels the raters can choose from is rather high (*k* equals approximately 200 to 300).