I have some content analysis data that consist of unordered mutually exclusive categories as rated by two coders.

What are useful approaches towards assessing inter-rater reliability for these data?

It occurs to me that failing to reject a $\chi^{2}$ test for association would be weak evidence of reliability, and that some kind of equivalence test for equivalence might be even stronger evidence of same (although I am not familiar with equivalence tests in a contingency table framework).

Other thoughts?

  • $\begingroup$ Have you considered using Cohen's Kappa? (What do you mean by equivalence tests in your situation?) Failing to reject a χ2 test for association would seem to provide NO evidence for reliability. $\endgroup$ – Joel W. Jan 15 '15 at 20:32
  • $\begingroup$ @JoelW. Cohen's Kappa applies to two categories, ostensibly ordered since such data typically indicate code present/absent. Cohen's Kappa has been extended to more than 2 raters (i.e. Fleiss Kappa), and been extended to an arbitrary number of ordered levels and even continuous data (i.e. Krippendorff's $\alpha$). But none of those describes applications for my data: mutually exclusive unordered categories. $\endgroup$ – Alexis Jan 15 '15 at 20:54
  • $\begingroup$ @JoelW. Equivalence tests != "failing to reject." From my limited understanding, contingency table equivalence tests rely on the idea of table "collapsability" by category. Something like: if one rejects a null hypothesis that categories cannot be collapsed (by some tolerance; i.e. that there is an association), then one would conclude that the distributions of categories are equivalent. That would seem to provide evidence of reliability. $\endgroup$ – Alexis Jan 15 '15 at 20:57
  • $\begingroup$ Sometimes Kappa is described as applicable to ordered data, but sometimes it is described as applicable to categorical data, such as here: psych.unl.edu/psycrs/handcomp/hckappa.PDF I am unfamiliar with contingency table equivalence tests. Do you have a reference that describes them? $\endgroup$ – Joel W. Jan 16 '15 at 0:55
  • $\begingroup$ @JoelW. TY for the ref. Here's the big kid's equivalence test text (heady for me): Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority. Chapman and Hall/CRC Press, second edition. $\endgroup$ – Alexis Jan 16 '15 at 3:25

Consider using Cohen's Kappa. Sometimes Kappa is described as applicable to ordered data, but sometimes it is described as applicable to categorical data, such as here: http://psych.unl.edu/psycrs/handcomp/hckappa.PDF. Also, this page discusses intercoder/interrater reliability coefficients for nominal, ordinal, interval, or ratio-level data: http://dfreelon.org/utils/recalfront

  • $\begingroup$ The original "unweighted" kappa coefficient is applicable to nominal categories only. However, later versions "weighted" versions enabled it to be applicable to ordered categories as well. $\endgroup$ – Jeffrey Girard Apr 14 '16 at 5:21

Try Krippendorff's alpha agreement measure which has nominal, ordinal, interval, and ratio variants, can handle any number of raters/coders, as well as missing values.

The citation below describes how to implement Krippendorff's alpha agreement coefficient in SPSS and SAS, but there are also packages for Stata and R.

Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77-89. https://doi.org/10.1080/19312450709336664

  • $\begingroup$ Hi Kostas. Wecome to CV. Can you explain how to address my question about equivalence? $\endgroup$ – Alexis Mar 3 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.