Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm trying to figure out what to use to calculate inter-rater reliability for a set of behavioral coding. There's one individual codes the whole set (primary coder; e.g. codes data ID 1-300) and a few other secondary coders who codes different sections of the data (Person A codes 1-100, Person B codes 101-200, Person C codes 202-300).
I'm not sure how to find the reliability in this case. I'm leaning towards separate Cohen's Kappa scores for each of the rater.
If you compute $\kappa$ for each of the three subsets you can then summarise it either by taking the median or by a weighted mean with weights the inverse of the sampling variances. That gives you the average agreement of a secondary rater with the chosen primary rater. If the primary rater is seen as a gold standard, perhaps because they are highly trained and the others are beginners, then computing sensitivity and specificity for each subset would seem a better option.
