I am trying to calculate inter-rater reliability for two raters using a dichotomous scale.
In one part of the study, raters have to decide from a list which word candidates of a child's utterances are real words. This works fine with Cohen's Kappa.
However, in a second part of the study, raters are asked to write down a list of real words a child produces, i.e. they don't choose from a list, but make their own. It seems I can't use Kappa for calculating agreement here (I get a negative value), and I assume it is because I only have 3 cells (word present(+rater1/+rater2), word present (+rater1/-rater2), word present (-rater1/+rater2), missing the cell where raters agree on a word being absent as in the previous part of the study. So my question is: how do I calculate inter-rater reliability in this case?