I have a set of n texts. Each text is to be assigned a topic by i raters who choose the topic from a set of k discrete topic labels (A, B, C, ... K). So, for each text there are i ratings (categorical data):

On this dataset, I want to calculate inter-rater agreement. Given that there are more than two raters, Fleiss' kappa would be a reasonable choice, since Cohen's kappa measures agreement between two raters only.

Given that both the texts to be annotated as well as the raters who annotate the texts are costly, I need to limit the number of texts or the number of raters; otherwise the study would be too expensive.

Now my question: In view of this restriction, which one (number of raters or number of texts) contributes more to the robustness of the inter-rater reliability measure? In other words, is it better to limit the number n of texts to be annotated, or the number i of raters if my goal is to assess how generalisable the obtained results are. The overall goal of the study is to assess how useful the set of k discrete topic labels is for the task of text annotation. Note that the both the raters and the texts in the study are sampled from a potentially very large population of texts/raters. Besides that, the number k of categorical topic labels the raters can choose from is rather high (k equals approximately 200 to 300).

Any statistical advice will be highly appreciated.

I have a set of n texts. Each text is to be assigned a topic by i raters who choose the topic from a set of k discrete topic labels (A, B, C, ... K). So, for each text there are i ratings (categorical data):

On this dataset, I want to calculate inter-rater agreement. Given that there are more than two raters, Fleiss' kappa would be a reasonable choice, since Cohen's kappa measures agreement between two raters only.

Given that both the texts to be annotated as well as the raters who annotate the texts are costly, I need to limit the number of texts or the number of raters; otherwise the study would be too expensive.

Now my question: In view of this restriction, which one (number of raters or number of texts) contributes more to the robustness of the inter-rater reliability measure? In other words, is it better to limit the number n of texts to be annotated, or the number i of raters if my goal is to assess how generalisable the obtained results are. The overall goal of the study is to assess how useful the set of k discrete topic labels is for the task of text annotation. Note that the both the raters and the texts in the study are sampled from a potentially very large population of texts/raters. Besides that, the number k of categorical topic labels the raters can choose from is rather high (k equals approximately 200 to 300).