Krippendorff's alpha in R for multi-label annotation

Question

I need to calculate inter-rater agreement (2 annotators) on a small dataset (let's say 10 items), where annotators used between 1 and 4 labels per item. The total number of different labels is pretty small (~5). As an example, let's say I have the following where lx are different labels:

       Coder1    Coder2
Item0  l1, l2    l1
Item1  l1, l2    l1, l2
Item2  l1        l1
Item3  l3        l3
Item4  l3        l1, l3
Item5  l4        l4
Item6  l2        l4
Item7  l1, l2    l1
Item8  l1, l2    l1, l2, l3
Item9  l1        l1
Item0  l1        l1

Based on everything I've read, Krippendorff's alpha seems to be the thing I should use, but I can't figure out how to use it with multi-label data instead of single-label data.

How can I measure inter-rater agreement with multi-label data?

Answer 1

Your question can be seen as a mix of two different questions. First one is how to calculate agreement in case of multi-label annotation. The second one is what function to use, in R or in a different language, to do this calculation.

1. How to calculate agreement for multi-label annotation.

The answer to first question is that you should:

• choose a weighted agreement coefficient, such as Krippendorff's Alpha or Fleiss Kappa (multi-kappa as defined in this comprehensive survey of agreement measures by Artstein and Poesio).
• choose how you calculate distance between two sets of labels. MASI distance is among the possible options.

2. How to calculate agreement for multi-label annotation in R or other language/tool

Here is explained how to calculate Krippendorff's alpha for multi-label annotation using MASI distance using Python.

Here is a working example with a part of your data:

import nltk
from nltk.metrics import agreement
from nltk.metrics.agreement import AnnotationTask
from nltk.metrics import masi_distance

task_data = [('coder1','Item0',frozenset(['l1','l2'])),
('coder2','Item0',frozenset(['l1'])),
('coder1','Item1',frozenset(['l1','l2'])),
('coder2','Item1',frozenset(['l1','l2'])),
('coder1','Item2',frozenset(['l1'])),
('coder2','Item2',frozenset(['l1']))]

task = AnnotationTask(distance = masi_distance)

task.load_array(task_data)

task.alpha()

I have not found any implementation Krippendorff's alpha in R that lets you use custom distance function as nltk.metrics.agreement does.

The solution is to use irrCAC library, as it's krippen.alpha function accepts the weight matrix as argument. Than, one can calculate the weight matrix for all observed combination of labels using any distance function. Here is my solution. I call python to calculate MASI distance.

library(irrCAC)
library(reticulate)

#creating the dataset as dataframe
dataset <- c("l1, l2",
             "l1","l1, l2",
             "l1, l2","l1",
             "l1","l3","l3",
             "l3","l1, l3","l4",
             "l4","l2",
             "l4","l1, l2",
             "l1","l1, l2",
             "l1, l2, l3",
             "l1","l1",
             "l1","l1")
dataset <- data.frame(matrix(dataset,ncol = 2, byrow = T), stringsAsFactors = F)
colnames(dataset) <- c("Coder1", "Coder2")

#function that calculates MASI distance using python nltk.metrics 

pybuiltins <- import_builtins()
nltk <- import("nltk")
masi <- function(x,y,split = " ")
{
  if(length(str_split(x,split,simplify = F)[[1]]) == 1)
  {
    x <- str_split(x,split,simplify = F)
  }else
  {
    x <- unlist(str_split(x,split,simplify = F))
  }
  
  if(length(str_split(y,split,simplify = F)[[1]]) == 1)
  {
    y <- str_split(y,split,simplify = F)
  }else
  {
    y <- unlist(str_split(y,split,simplify = F))
  }
  
  masiD <-   nltk$masi_distance(pybuiltins$set(x),pybuiltins$set(y))
  masiD  
}

#calculating weight matrix for all observed combination of lables using MASI distance
labelsCombAll <- unique(unlist(dataset))
distMasi <- matrix(nrow = length(labelsCombAll),ncol = length(labelsCombAll),
                   dimnames = list(labelsCombAll,labelsCombAll))

for(i in 1:nrow(distMasi))
  for(j in 1:ncol(distMasi))
  {
    distMasi[i,j] <- 1-masi(labelsCombAll[i],labelsCombAll[j],", ")
  }

#calculating krippendorff alpha
krippen.alpha.raw(dataset, weights = distMasi,categ.labels = rownames(distMasi))
```