Context: I used Fleiss Kappa to compute inter-rater reliability for categorical judgements on a list of words. The raters were given instructions and asked to judge whether a list of ~3700 words were events words. The final data frame ends up looking as below where 1 = event and 2 = not event.
word judge1 judge 2 judge3
x 0 1 1
y 0 0 0
z 1 1 1
Output: Cohen's kappa was computed to assess the agreement between 3 graduate researchers in judging words as event words. There was poor agreement beyond chance between the 3 judges, kappa = -0.11, p < .05.
Situation: If I am correct, the kappa statistic is showing that they performed worse than could be by chance (possibly due to the long list?). I am wondering if there's anything I can do from here? Get more people to judge the long list?