2
$\begingroup$

I have some content analysis data that consist of unordered mutually exclusive categories as rated by two coders.

What are useful approaches towards assessing inter-rater reliability for these data?

It occurs to me that failing to reject a $\chi^{2}$ test for association would be weak evidence of reliability, and that some kind of equivalence test for equivalence might be even stronger evidence of same (although I am not familiar with equivalence tests in a contingency table framework).

Other thoughts?

Improve this question
$\endgroup$
9
  • $\begingroup$ Have you considered using Cohen's Kappa? (What do you mean by equivalence tests in your situation?) Failing to reject a χ2 test for association would seem to provide NO evidence for reliability. $\endgroup$
    – Joel W.
    Commented Jan 15, 2015 at 20:32
  • $\begingroup$ @JoelW. Cohen's Kappa applies to two categories, ostensibly ordered since such data typically indicate code present/absent. Cohen's Kappa has been extended to more than 2 raters (i.e. Fleiss Kappa), and been extended to an arbitrary number of ordered levels and even continuous data (i.e. Krippendorff's $\alpha$). But none of those describes applications for my data: mutually exclusive unordered categories. $\endgroup$
    – Alexis
    Commented Jan 15, 2015 at 20:54
  • $\begingroup$ @JoelW. Equivalence tests != "failing to reject." From my limited understanding, contingency table equivalence tests rely on the idea of table "collapsability" by category. Something like: if one rejects a null hypothesis that categories cannot be collapsed (by some tolerance; i.e. that there is an association), then one would conclude that the distributions of categories are equivalent. That would seem to provide evidence of reliability. $\endgroup$
    – Alexis
    Commented Jan 15, 2015 at 20:57
  • $\begingroup$ Sometimes Kappa is described as applicable to ordered data, but sometimes it is described as applicable to categorical data, such as here: psych.unl.edu/psycrs/handcomp/hckappa.PDF I am unfamiliar with contingency table equivalence tests. Do you have a reference that describes them? $\endgroup$
    – Joel W.
    Commented Jan 16, 2015 at 0:55
  • $\begingroup$ @JoelW. TY for the ref. Here's the big kid's equivalence test text (heady for me): Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority. Chapman and Hall/CRC Press, second edition. $\endgroup$
    – Alexis
    Commented Jan 16, 2015 at 3:25

2 Answers 2

Reset to default
2
$\begingroup$

Consider using Cohen's Kappa. Sometimes Kappa is described as applicable to ordered data, but sometimes it is described as applicable to categorical data, such as here: http://psych.unl.edu/psycrs/handcomp/hckappa.PDF. Also, this page discusses intercoder/interrater reliability coefficients for nominal, ordinal, interval, or ratio-level data: http://dfreelon.org/utils/recalfront

Improve this answer
$\endgroup$
1
  • $\begingroup$ The original "unweighted" kappa coefficient is applicable to nominal categories only. However, later versions "weighted" versions enabled it to be applicable to ordered categories as well. $\endgroup$
    – Jeffrey Girard
    Commented Apr 14, 2016 at 5:21
0
$\begingroup$

Try Krippendorff's alpha agreement measure which has nominal, ordinal, interval, and ratio variants, can handle any number of raters/coders, as well as missing values.

The citation below describes how to implement Krippendorff's alpha agreement coefficient in SPSS and SAS, but there are also packages for Stata and R.

Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77-89. https://doi.org/10.1080/19312450709336664

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi Kostas. Wecome to CV. Can you explain how to address my question about equivalence? $\endgroup$
    – Alexis
    Commented Mar 3, 2019 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.