Intercoder reliability without mutually-exclusive codes

Question

I have the results from qualitatively coding the contents of ~100 images, with 7 categories and 96 tags in total. It's incredibly difficult to define mutually exclusive codes for the content we're interested in. For example, we code whether a person is depicted acting as a consumer, as an employee, and/or as a family member. With so many codes, it's a nightmare to define hyper-specific categories that would guarantee mutual exclusivity of every code.

Q: Are there any methods for estimating inter-rater reliability for data that violates the assumption of mutual exclusivity?

Answer 1

Krippendorff (known for his, extremely flexible, inter-rate agreement coefficient) has come up with a proposal on how to measure reliability when raters have multiple interpretations of the same unit. However, I do not know any statistical package that implements this idea.

Krippendorff, K. (2004). Measuring the Reliability of Qualitative Text Analysis Data. Quality and Quantity, 38 (6), 787-800. https://doi.org/10.1007/s11135-004-8107-7

Answer 2

It turns out that no, there isn't any well-established way of solving this problem. One name for this type of coding is "one-to-many classification".

However, Kirilenko and Svetlana published an article last year on a "fuzzy kappa" that addresses this problem. They also released demonstration software (available as an executable on SourgeForge). That said, I couldn't actually find any papers or projects that actually used this test. They do nicely cover the problem of mutually exclusive codes, saying,

The existing methods of agreement estimation, e.g., Cohen’s kappa, require that coders place each unit of content into one and only one category (one-to-one coding) from the pre-established set of categories. However, in certain data domains (e.g., maps, photographs, databases of texts and images), this requirement seems overly restrictive.

Kirilenko, Andrei P., and Svetlana Stepchenkova. "Inter-coder agreement in one-to-many classification: Fuzzy kappa." PloS one 11.3 (2016): e0149787.