Inter-rater agreement with different rater pairings

I have data from the following study design:

100 Patients have been examined by 2 raters each. The raters had to assign a score which consists of 7 subscores and each of this subscore consists of various items. In total there are around 100 items per patient and examination. The scale of the items is discrete, ranging from 0 to 3 or up to 7. So, I would analyse them as ordinal data. There were a total of 4 raters, resulting in 6 possible rater-pairings. Each patient was examined by 1 rater-pairing. Raters differ in the number of patients they examined, so some rater-pairings will be more common than others. The aim is to quantify which of these 100 items, from which the final score is calculated, show the biggest "inconsistency" between raters.

I would like to know the following:

1. Which measure is appropriate for quantifying rater agreement for this data structure? I am leaning towards the weighted kappa, but unsure whether this is correct.

2. How to deal with the fact that there are different rater-pairs? Is there a way to combine them, thus averaging over all rater-pairs, maybe by somehow weighting the rater-pairs by the number of patients they examined? That would give me some kind of overall-agreement.

The difference between rater-pairings is not really of interest. Although I think, if necessary, I could split the data set into rater-pair-specific subsets and analyse them separately.

I am working with R. But since I assume that all relevant measures are already implemented (I know that "irr" can calculate a load of different coefficients) I am not worried about that. I would rather know how to apply which measure.

Edited: Here is some R code for toy data:

dat.ex1 <- data.frame(examination=rep(1,5),patient=1:5,rater=sample(c("A","B","C",
"D"),5,replace=TRUE),item1=sample(0:4,5,replace=TRUE),item2=sample(0:4,5,replace=
TRUE),item3=sample(0:4,5,replace=TRUE),item4=sample(0:4,5,replace=TRUE),item5=sam
ple(0:4,5,replace=TRUE))

dat.ex2 <- data.frame(examination=rep(2,5),patient=1:5,rater=c("B","C","B","B","D")
,item1=c(3,4,3,1,1),item2=c(1,3,0,0,3),item3=c(4,0,0,3,3),item4=c(1,0,1,0,1),item5=
c(4,2,3,1,0))


Thus, I have the data in 2 data files: one for the first examination, one for the second examination. In each file, there is 1 row per patient. For example, patient 1 has first been examined by rater C who rated him 4 for item 1, then the patient was examined by rater B, who rated him 3 for item 1. In the toy data, as in the real data, the number of ratings differed between raters (A:2,B:3,C:4,D:1), and accordingly, the frequency of rater-pairs differs (here AB:0,AC:1,AD:1,BC:3,BD:0,CD:0).

In total there are 100 patients and 102 items.

The aim is to select those items that show a high disagreement between raters, because these would be the items that need better instructions, etc.

• I'd be really helpful if you could provide some (simulated) toy-data that shows the setup. A quick suggestion from my side is to employ a variant of weighted Fleiss' kappa. – Anders Ellern Bilgrau Aug 15 '15 at 16:33
• @AEBilgrau: Thank you for your comment. I added some R code for similar toy data above and tried to describe the data structure in more detail. I also came about Fleiss' kappa, but the problem is that there are always only 2 raters per patient and Fleiss' kappa is recommended when there are >2 raters per subject. WHat exactly did you mean with "a variant of weighted Fleiss' kappa"? – Deborah Vogt Aug 15 '15 at 21:05
• @DeborahVogt: Two alternatives for the output: 1.) You can take a screengrab of the output and post it as a picture in your post, or 2.) Copy and paste the output into the text editor field of your posting, highlight the output, and click on the "code" button in the formatting toolbar (the one that looks like a set of curly brackets { } ). HTH – Marquis de Carabas Aug 15 '15 at 21:46
• By weighted Fleiss' kappa I mean to use the weights to describe the ordinal nature of the items. I.e. you create a distance matrix that describes the distance (weight) between item i and item j. The variant of was aimed the multiple rater problem and intentionally vague (because I don't know the answer). But I guess you need to come up with a "average" inter-rater correlation, as you say yourself. This could e.g. be by the average of all pairwise rater combinations. To my knowledge, there is no standard way of doing it. – Anders Ellern Bilgrau Aug 16 '15 at 10:51