# Inter-rater reliability for unordered categorical data

I have some content analysis data that consist of unordered mutually exclusive categories as rated by two coders.

What are useful approaches towards assessing inter-rater reliability for these data?

It occurs to me that failing to reject a $\chi^{2}$ test for association would be weak evidence of reliability, and that some kind of equivalence test for equivalence might be even stronger evidence of same (although I am not familiar with equivalence tests in a contingency table framework).

Other thoughts?

• Have you considered using Cohen's Kappa? (What do you mean by equivalence tests in your situation?) Failing to reject a χ2 test for association would seem to provide NO evidence for reliability. – Joel W. Jan 15 '15 at 20:32
• @JoelW. Cohen's Kappa applies to two categories, ostensibly ordered since such data typically indicate code present/absent. Cohen's Kappa has been extended to more than 2 raters (i.e. Fleiss Kappa), and been extended to an arbitrary number of ordered levels and even continuous data (i.e. Krippendorff's $\alpha$). But none of those describes applications for my data: mutually exclusive unordered categories. – Alexis Jan 15 '15 at 20:54
• @JoelW. Equivalence tests != "failing to reject." From my limited understanding, contingency table equivalence tests rely on the idea of table "collapsability" by category. Something like: if one rejects a null hypothesis that categories cannot be collapsed (by some tolerance; i.e. that there is an association), then one would conclude that the distributions of categories are equivalent. That would seem to provide evidence of reliability. – Alexis Jan 15 '15 at 20:57
• Sometimes Kappa is described as applicable to ordered data, but sometimes it is described as applicable to categorical data, such as here: psych.unl.edu/psycrs/handcomp/hckappa.PDF I am unfamiliar with contingency table equivalence tests. Do you have a reference that describes them? – Joel W. Jan 16 '15 at 0:55
• @JoelW. TY for the ref. Here's the big kid's equivalence test text (heady for me): Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority. Chapman and Hall/CRC Press, second edition. – Alexis Jan 16 '15 at 3:25