I am trying to calculate inter-rater reliability for two raters using a dichotomous scale.

In one part of the study, raters have to decide from a list which word candidates of a child's utterances are real words. This works fine with Cohen's Kappa.

However, in a second part of the study, raters are asked to write down a list of real words a child produces, i.e. they don't choose from a list, but make their own. It seems I can't use Kappa for calculating agreement here (I get a negative value), and I assume it is because I only have 3 cells (word present(+rater1/+rater2), word present (+rater1/-rater2), word present (-rater1/+rater2), missing the cell where raters agree on a word being absent as in the previous part of the study. So my question is: how do I calculate inter-rater reliability in this case?

  • 1
    $\begingroup$ No need to apologise. If your question is good, it will get answered. If not, we won't treat you differently because of an apology! I think the underlying question here is what you really want to measure. Let's guess that there are thousands and thousands of words that neither of us used in a conversation today. How do you even list them? Is that relevant to comparing our vocabularies? I'd look at measures used in cluster analysis, e.g. Jaccard coefficient, #words in common/#words used by either. More deeply, except that I can't oblige, there should be literature in your field on this. $\endgroup$ – Nick Cox Mar 23 '17 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.