I have done a test where 2 different raters had to rate 1000 texts independently according to the following options:

  • Happy
  • Sad
  • Angry
  • Confused
  • Could not tell

Now, I am trying to determine their agreement (i.e reliability). However, I am facing a dilemma -specially when the "could not tell" option has been selected. In particular, my question is shall I include the "could not tell" option for the reliability calculation by using Krippendorff’s Alpha OR should I remove it from my analysis as the "could not tell" option can't be treated as disagreement?

Thank you in advance.


You could consider "Could not tell" as a fifth and separate category, in which case you would just leave it in there (this is the direction I would lean toward). Or you could consider it as missing data, in which case you would exclude it from agreement analysis but still report how often it occurred. Another approach would be to calculate and report agreement both ways so that readers of your work will know how sensitive your results are to this choice. If both are nearly the same, then it doesn't really matter which you use.

  • 1
    $\begingroup$ Phenomenal response, highly appreciate it ! thank you!! $\endgroup$
    – Louise
    Apr 28 at 8:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.