# Inter-rater reliability for binomial repeated ratings from two or more raters

I would like to calculate the inter-rater reliability for a binomial rating (0 or 1) from multiple events from different subjects by two raters.

Here are some data in R:

 Subject <- c(1, 2, 2, 3, 4, 4, 4, 4, 5, 6, 6, 7, 8, 8, 8, 8, 8, 8, 9, 9,
10, 10, 10, 11, 11)
Subject <- c(Subject, Subject)
Rating <- ifelse(Subject %% 2 == 1, 0, 1)
set.seed(1)
index <- sample(1:length(Subject), 8,1)
Rating[index] <- 1
Rater <- rep(1:2, each = 25)

data <- data.frame(Rating, Subject, Rater)
head(data) # show the first six rows tail(data) # show the last six rows Now I could calculate for example Krippendorfs alpha:

library(irr)

data2 <- data
data2$No <- rep(1:25, 2) library(reshape2) datawide <- dcast(data2, No + Subject ~ Rater, value.var= "Rating") datawide$No <- NULL
datawide\$Subject <- NULL

datawide <- as.matrix(t(datawide)) kripp.alpha(datawide, method = "nominal")


Krippendorfs alpha = 0.57.

The problem is that the number of events varies from subject to subject and thus, the events are not independent from each other. Therfore, I can not just compute Krippendorfs alpha or any other measure of IRR that I know (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/ ).

Does anyone know how I could calculate an inter-rater reliability in this setting?