I need to calculate inter-rater agreement (2 annotators) on a small dataset (let's say 10 items), where annotators used between 1..4 labels per item. The total number of different labels is pretty small (~5). As an example, let's say I have the following where lx are different labels:

       Coder1    Coder2
Item0  l1, l2    l1
Item1  l1, l2    l1, l2
Item2  l1        l1
Item3  l3        l3
Item4  l3        l1, l3
Item5  l4        l4
Item6  l2        l4
Item7  l1, l2    l1
Item8  l1, l2    l1, l2, l3
Item9  l1        l1
Item0  l1        l1

Based on everything I've read, Krippendorff's alpha seems to be the thing I should use, but I can't figure out how to get this kind of data into R's kripp.alpha function. All the examples (e.g., https://www.rdocumentation.org/packages/irr/versions/0.84.1/topics/kripp.alpha) seem to assume 0..1 labels per item. No need to use R for this, if somebody has a good solution in a different language or tool, I'm happy to switch.


Your question can be seen as a mix of two different questions. First one is how to calculate agreement in case of multi-label annotation. The second one is what function to use, in R or in a different language, to do this calculation.

  1. How to calculate agreement for multi-label annotation.

The answer to first question is that you should:

  1. How to calculate agreement for multi-label annotation in R or other language/tool

Here is explained how to calculate Krippendorff's alpha for multi-label annotation using MASI distance using Python.

Here is a working example with a part of your data:

import nltk
from nltk.metrics import agreement
from nltk.metrics.agreement import AnnotationTask
from nltk.metrics import masi_distance

task_data = [('coder1','Item0',frozenset(['l1','l2'])),

task = AnnotationTask(distance = masi_distance)



I have not found any implementation Krippendorff's alpha in R that lets you use custom distance function as nltk.metrics.agreement does.

The solution is to use irrCAC library, as it's krippen.alpha function accepts the weight matrix as argument. Than, one can calculate the weight matrix for all observed combination of labels using any distance function. Here is my solution. I call python to calculate MASI distance.


#creating the dataset as dataframe
dataset <- c("l1, l2",
             "l1","l1, l2",
             "l1, l2","l1",
             "l3","l1, l3","l4",
             "l4","l1, l2",
             "l1","l1, l2",
             "l1, l2, l3",
dataset <- data.frame(matrix(dataset,ncol = 2, byrow = T), stringsAsFactors = F)
colnames(dataset) <- c("Coder1", "Coder2")

#function that calculates MASI distance using python nltk.metrics 

pybuiltins <- import_builtins()
nltk <- import("nltk")
masi <- function(x,y,split = " ")
  if(length(str_split(x,split,simplify = F)[[1]]) == 1)
    x <- str_split(x,split,simplify = F)
    x <- unlist(str_split(x,split,simplify = F))
  if(length(str_split(y,split,simplify = F)[[1]]) == 1)
    y <- str_split(y,split,simplify = F)
    y <- unlist(str_split(y,split,simplify = F))
  masiD <-   nltk$masi_distance(pybuiltins$set(x),pybuiltins$set(y))

#calculating weight matrix for all observed combination of lables using MASI distance
labelsCombAll <- unique(unlist(dataset[,2:ncol(dataset)]))
distMasi <- matrix(nrow = length(labelsCombAll),ncol = length(labelsCombAll),
                   dimnames = list(labelsCombAll,labelsCombAll))

for(i in 1:nrow(distMasi))
  for(j in 1:ncol(distMasi))
    distMasi[i,j] <- 1-masi(labelsCombAll[i],labelsCombAll[j],", ")

#calculating krippendorff alpha
krippen.alpha.raw(dataset, weights = distMasi,categ.labels = rownames(distMasi))
| cite | improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.