Inter-rater reliability for binomial repeated ratings from two or more raters

Question

I would like to calculate the inter-rater reliability for a binomial rating (0 or 1) from multiple events from different subjects by two raters.

Here are some data in R:

 Subject <- c(1, 2, 2, 3, 4, 4, 4, 4, 5, 6, 6, 7, 8, 8, 8, 8, 8, 8, 9, 9, 
 10, 10, 10, 11, 11)
 Subject <- c(Subject, Subject)
 Rating <- ifelse(Subject %% 2 == 1, 0, 1)
 set.seed(1)
 index <- sample(1:length(Subject), 8,1)
 Rating[index] <- 1
 Rater <- rep(1:2, each = 25)


 data <- data.frame(Rating, Subject, Rater)
 head(data) # show the first six rows

 tail(data) # show the last six rows

Now I could calculate for example Krippendorfs alpha:

library(irr)

data2 <- data
data2$No <- rep(1:25, 2)

library(reshape2)

datawide <- dcast(data2, No + Subject ~ Rater, value.var= "Rating")
datawide$No <- NULL
datawide$Subject <- NULL

datawide <- as.matrix(t(datawide))

kripp.alpha(datawide, method = "nominal")

Krippendorfs alpha = 0.57.

The problem is that the number of events varies from subject to subject and thus, the events are not independent from each other. Therfore, I can not just compute Krippendorfs alpha or any other measure of IRR that I know (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/ ).

Does anyone know how I could calculate an inter-rater reliability in this setting?

Answer 1

Klaus Krippendorff just confirmed me via email that I do can perform the analysis exactly as I did. Here is his answer:

"as long as you can distinguish one event from another and ask your two observers to categorize, rate or value them, your units are not subjects but events. the unequal number of events per subject, and the likelihood that a 2nd events is qualitatively or quantitatively related to the 1st is something that your analysis of the data will reveal. it seems to me you have a data structure like:

with cells occupied by categories, measures of intensity or other values. this would get you to the publishable reliability of the two observer’s accounts of events"

Thank you very much!