I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each subject was rated by two coders. Lately, I wanted to make a start with assessing the IRR using Kappa Statistics. To do so, I structured the data with separate variables for each coder (see the print screen below). However, at the moment I wanted to calculate the Kappa Statistic, I discovered that a Kappa Statistics can only be calculated when all the ratings from the different coders are merged in two columns (i.e. one column for rater 1 and one column for rater 2). Since each subject is rated by a different set of raters, my dataset contains more than 2 columns, which hinders me to calculate a Kappa Statistic for the variable of interest. I searched for a solution, but unfortunately couldn’t find one. My basic question therefore is: how to calculate a Kappa Statistic in a dataset like mine? Would it make sense to just merge all the ratings in two columns or is there a better solution?
Any advice will be highly appreciated!