# Computing Inter-Rater Reliability - Different subjects are rated by different subsets of coders

I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each subject was rated by two coders. Lately, I wanted to make a start with assessing the IRR using Kappa Statistics. To do so, I structured the data with separate variables for each coder (see the print screen below). However, at the moment I wanted to calculate the Kappa Statistic, I discovered that a Kappa Statistics can only be calculated when all the ratings from the different coders are merged in two columns (i.e. one column for rater 1 and one column for rater 2). Since each subject is rated by a different set of raters, my dataset contains more than 2 columns, which hinders me to calculate a Kappa Statistic for the variable of interest. I searched for a solution, but unfortunately couldn’t find one. My basic question therefore is: how to calculate a Kappa Statistic in a dataset like mine? Would it make sense to just merge all the ratings in two columns or is there a better solution?

Any advice will be highly appreciated!

The original kappa formula (and that used by SPSS) converts the data into a $q$-by-$q$ confusion matrix (item counts in a category-by-category matrix). This formula required exactly two raters. Later "extended" formulas convert the data into an $n$-by-$q$ classification matrix (rater counts in an item-by-category matrix). This formulation allowed any number of raters and various patterns of missingness. The intuition here is that with $X$ raters rating a given item, there are $X(X-1)$ possible agreements.
• In case you don't know, there are math typesetting tools on this site that, so that for instance $x$ renders as $x$. There are some editing tips here. – Silverfish Apr 9 '16 at 14:19