0
$\begingroup$

Context: I used Fleiss Kappa to compute inter-rater reliability for categorical judgements on a list of words. The raters were given instructions and asked to judge whether a list of ~3700 words were events words. The final data frame ends up looking as below where 1 = event and 2 = not event.


word judge1  judge 2  judge3 
x    0          1        1
y    0          0        0
z    1          1        1

Output: Cohen's kappa was computed to assess the agreement between 3 graduate researchers in judging words as event words. There was poor agreement beyond chance between the 3 judges, kappa = -0.11, p < .05.

Situation: If I am correct, the kappa statistic is showing that they performed worse than could be by chance (possibly due to the long list?). I am wondering if there's anything I can do from here? Get more people to judge the long list?

Improve this question
$\endgroup$
4
  • $\begingroup$ It would be helpful if there was a gold standard against which you could compare results -- i.e., a "truth table". For example, there are dictionaries for good and bad words, which are employed for sentiment mining (restaurant ratings, on-line reviews). At present, you know the judges don't rate the same, but you don't know which if any judge is way off on whether a word truthfully represents an event or not. $\endgroup$
    – user318288
    Aug 15, 2021 at 15:35
  • $\begingroup$ I agree @user0123456789. I haven't found a dictionary for event words, but at the same time, I hardly know where to look. Would you have an idea of where to look to find this? $\endgroup$
    – n.baes
    Aug 15, 2021 at 23:39
  • $\begingroup$ I should add that had I found such a list earlier, people wouldn't have needed to judge the words as event words or not. $\endgroup$
    – n.baes
    Aug 15, 2021 at 23:42
  • $\begingroup$ I'm not familiar with event words, but would suggest looking around for what users of Atlas.ti use for such analyses. I've known psychometricians who've used Atlas.ti for hunting down key words in spoken comments/phrases from attendees at focus groups (cancer patients). $\endgroup$
    – user318288
    Aug 17, 2021 at 2:18

1 Answer 1

Reset to default
1
$\begingroup$

Chance-corrected interrater agreement coefficients (Krippendorff's Alpha, Scott's Pi, Cohen's Kappa) are very conservative if the distribution of your judgements is biased. If there are 90% cases where 1 is chosen and only 10% with 0, you get a per-chance agreement of 82%. Only if your coders agree in more than 82%, will these coefficients be positive. (If you have a 99%-1% bias, you need more than 98% agreement).

In such cases, it is worthwile to report not just Cohen's Kappa but also some other coefficients that might help identify the problem:

  • Brennan & Prediger's Kappa: Uses a chance agreement that would be achieved by Laplace random choices (flipping a coin). Here, the chance agreement of coders on a 2-category variable is always 50%. Any agreement above is positive.
  • Precision and Recall: Especially in dichotomous cases, precision and recall are informative. These coefficients just indicate how many of the true 1 were found and how reliable the coding of 1 was. For these coeffieicents, you will need a gold standard, however.
  • Pairwise confusion matrices: Create cross-tables for each pair of coders to identify cases in which they deviated and find systematic errors (one coder that misses lots of 1 or 0). This might help in re-training the coders for subsequent analyses.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.