1
$\begingroup$

I'm trying to figure out what to use to calculate inter-rater reliability for a set of behavioral coding. There's one individual codes the whole set (primary coder; e.g. codes data ID 1-300) and a few other secondary coders who codes different sections of the data (Person A codes 1-100, Person B codes 101-200, Person C codes 202-300).

I'm not sure how to find the reliability in this case. I'm leaning towards separate Cohen's Kappa scores for each of the rater.

$\endgroup$
1
  • $\begingroup$ See chapter 11 "Inter-Rater Reliability: Conditional Analysis" in Gwet (2014) for a discussion of this type of analysis. Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advanced Analytics. $\endgroup$ Apr 27, 2021 at 4:07

1 Answer 1

1
$\begingroup$

If you compute $\kappa$ for each of the three subsets you can then summarise it either by taking the median or by a weighted mean with weights the inverse of the sampling variances. That gives you the average agreement of a secondary rater with the chosen primary rater. If the primary rater is seen as a gold standard, perhaps because they are highly trained and the others are beginners, then computing sensitivity and specificity for each subset would seem a better option.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.