Low inter-rater reliability for Fleiss Kappa

Context: I used Fleiss Kappa to compute inter-rater reliability for categorical judgements on a list of words. The raters were given instructions and asked to judge whether a list of ~3700 words were events words. The final data frame ends up looking as below where 1 = event and 2 = not event.


word judge1  judge 2  judge3
x    0          1        1
y    0          0        0
z    1          1        1


Output: Cohen's kappa was computed to assess the agreement between 3 graduate researchers in judging words as event words. There was poor agreement beyond chance between the 3 judges, kappa = -0.11, p < .05.

Situation: If I am correct, the kappa statistic is showing that they performed worse than could be by chance (possibly due to the long list?). I am wondering if there's anything I can do from here? Get more people to judge the long list?

• It would be helpful if there was a gold standard against which you could compare results -- i.e., a "truth table". For example, there are dictionaries for good and bad words, which are employed for sentiment mining (restaurant ratings, on-line reviews). At present, you know the judges don't rate the same, but you don't know which if any judge is way off on whether a word truthfully represents an event or not.
– user318288
Aug 15, 2021 at 15:35
• I agree @user0123456789. I haven't found a dictionary for event words, but at the same time, I hardly know where to look. Would you have an idea of where to look to find this? Aug 15, 2021 at 23:39
• I should add that had I found such a list earlier, people wouldn't have needed to judge the words as event words or not. Aug 15, 2021 at 23:42
• I'm not familiar with event words, but would suggest looking around for what users of Atlas.ti use for such analyses. I've known psychometricians who've used Atlas.ti for hunting down key words in spoken comments/phrases from attendees at focus groups (cancer patients).
– user318288
Aug 17, 2021 at 2:18

Chance-corrected interrater agreement coefficients (Krippendorff's Alpha, Scott's Pi, Cohen's Kappa) are very conservative if the distribution of your judgements is biased. If there are 90% cases where 1 is chosen and only 10% with 0, you get a per-chance agreement of 82%. Only if your coders agree in more than 82%, will these coefficients be positive. (If you have a 99%-1% bias, you need more than 98% agreement).
• Precision and Recall: Especially in dichotomous cases, precision and recall are informative. These coefficients just indicate how many of the true 1 were found and how reliable the coding of 1 was. For these coeffieicents, you will need a gold standard, however.
• Pairwise confusion matrices: Create cross-tables for each pair of coders to identify cases in which they deviated and find systematic errors (one coder that misses lots of 1 or 0). This might help in re-training the coders for subsequent analyses.