I have data from the following study design:
100 Patients have been examined by 2 raters each. The raters had to assign a score which consists of 7 subscores and each of this subscore consists of various items. In total there are around 100 items per patient and examination. The scale of the items is discrete, ranging from 0 to 3 or up to 7. So, I would analyse them as ordinal data. There were a total of 4 raters, resulting in 6 possible rater-pairings. Each patient was examined by 1 rater-pairing. Raters differ in the number of patients they examined, so some rater-pairings will be more common than others. The aim is to quantify which of these 100 items, from which the final score is calculated, show the biggest "inconsistency" between raters.
I would like to know the following:
Which measure is appropriate for quantifying rater agreement for this data structure? I am leaning towards the weighted kappa, but unsure whether this is correct.
How to deal with the fact that there are different rater-pairs? Is there a way to combine them, thus averaging over all rater-pairs, maybe by somehow weighting the rater-pairs by the number of patients they examined? That would give me some kind of overall-agreement.
The difference between rater-pairings is not really of interest. Although I think, if necessary, I could split the data set into rater-pair-specific subsets and analyse them separately.
I am working with R. But since I assume that all relevant measures are already implemented (I know that "irr" can calculate a load of different coefficients) I am not worried about that. I would rather know how to apply which measure.
Edited: Here is some R code for toy data:
dat.ex1 <- data.frame(examination=rep(1,5),patient=1:5,rater=sample(c("A","B","C", "D"),5,replace=TRUE),item1=sample(0:4,5,replace=TRUE),item2=sample(0:4,5,replace= TRUE),item3=sample(0:4,5,replace=TRUE),item4=sample(0:4,5,replace=TRUE),item5=sam ple(0:4,5,replace=TRUE)) dat.ex2 <- data.frame(examination=rep(2,5),patient=1:5,rater=c("B","C","B","B","D") ,item1=c(3,4,3,1,1),item2=c(1,3,0,0,3),item3=c(4,0,0,3,3),item4=c(1,0,1,0,1),item5= c(4,2,3,1,0))
Thus, I have the data in 2 data files: one for the first examination, one for the second examination. In each file, there is 1 row per patient. For example, patient 1 has first been examined by rater C who rated him 4 for item 1, then the patient was examined by rater B, who rated him 3 for item 1. In the toy data, as in the real data, the number of ratings differed between raters (A:2,B:3,C:4,D:1), and accordingly, the frequency of rater-pairs differs (here AB:0,AC:1,AD:1,BC:3,BD:0,CD:0).
In total there are 100 patients and 102 items.
The aim is to select those items that show a high disagreement between raters, because these would be the items that need better instructions, etc.