2
$\begingroup$

I would like to calculate the inter-rater reliability for a binomial rating (0 or 1) from multiple events from different subjects by two raters.

Here are some data in R:

 Subject <- c(1, 2, 2, 3, 4, 4, 4, 4, 5, 6, 6, 7, 8, 8, 8, 8, 8, 8, 9, 9, 
 10, 10, 10, 11, 11)
 Subject <- c(Subject, Subject)
 Rating <- ifelse(Subject %% 2 == 1, 0, 1)
 set.seed(1)
 index <- sample(1:length(Subject), 8,1)
 Rating[index] <- 1
 Rater <- rep(1:2, each = 25)


 data <- data.frame(Rating, Subject, Rater)
 head(data) # show the first six rows

enter image description here

 tail(data) # show the last six rows

enter image description here

Now I could calculate for example Krippendorfs alpha:

library(irr)

data2 <- data
data2$No <- rep(1:25, 2)

library(reshape2)

datawide <- dcast(data2, No + Subject ~ Rater, value.var= "Rating")
datawide$No <- NULL
datawide$Subject <- NULL

datawide <- as.matrix(t(datawide))

enter image description here

kripp.alpha(datawide, method = "nominal")

Krippendorfs alpha = 0.57.

The problem is that the number of events varies from subject to subject and thus, the events are not independent from each other. Therfore, I can not just compute Krippendorfs alpha or any other measure of IRR that I know (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/ ).

Does anyone know how I could calculate an inter-rater reliability in this setting?

$\endgroup$
1
$\begingroup$

Klaus Krippendorff just confirmed me via email that I do can perform the analysis exactly as I did. Here is his answer:

"as long as you can distinguish one event from another and ask your two observers to categorize, rate or value them, your units are not subjects but events. the unequal number of events per subject, and the likelihood that a 2nd events is qualitatively or quantitatively related to the 1st is something that your analysis of the data will reveal. it seems to me you have a data structure like:

enter image description here

with cells occupied by categories, measures of intensity or other values. this would get you to the publishable reliability of the two observer’s accounts of events"

Thank you very much!

$\endgroup$
  • $\begingroup$ This methods can be used for all metrics, with any number of observers and including missing data [Krippendorff, 2011] (repository.upenn.edu/cgi/…) $\endgroup$ – Kev Jan 15 '17 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.