Interrater reliability with multi-rater, multi-label dataset I'm trying to calculate interrater reliability with 3 coders, and each question is multiple choice. An example tagging would be:
QUESTION: What is the theme of this video?

VIDEO_0:

 - Coder0: food, sports
 - Coder1: food
 - Coder2: sports, education, music

VIDEO_1:

 - Coder0: sports
 - Coder1: sports
 - Coder2: sports, education

VIDEO_i etc.


After some reading,  I saw that Fleiss's Kappa and Krippendorff's Alpha are the two most common metric to calculate multiple label dataset. The following is my code:
import nltk
from nltk.metrics import agreement,masi_distance,jaccard_distance
from nltk.metrics.agreement import AnnotationTask

task_data = [('coder1','video0',frozenset(['food','sports'])),
            ('coder2','video0',frozenset(['food'])),
            ('coder3','video0',frozenset(['sports', 'education', 'music'])),
            ('coder1','video1',frozenset(['sports'])),
            ('coder2','video1',frozenset(['sports'])),
            ('coder3','video1',frozenset(['sports', 'education']))]

jaccard_task = AnnotationTask(data=task_data,distance = jaccard_distance)
masi_task = AnnotationTask(data=task_data,distance = masi_distance)

print(f"Fleiss's Kappa using Jaccard: {jaccard_task.multi_kappa()}")
print(f"Fleiss's Kappa using MASI: {masi_task.multi_kappa()}")
print(f"Krippendorff's Alpha using Jaccard: {jaccard_task.alpha()}")
print(f"Krippendorff's Alpha using MASI: {masi_task.alpha()}")

And got
Fleiss's Kappa using Jaccard: 0.4090909090909091
Fleiss's Kappa using MASI: 0.28863636363636364
Krippendorff's Alpha using Jaccard: 0.15217391304347838
Krippendorff's Alpha using MASI: 0.12971750574627405

My questions are

*

*Why is Jaccard and MASI distance would result in such significant difference?

*Are there better score for this type of dataset?

 A: I've decided to go with Kleiss's Kappa using Cosine distance (+sklearn.MultiLabelBinarizer) and Jaccard distance. My simple pipeline for Cosine distance for reference; the Jaccard version is simpler and referenced above.
from nltk.metrics import agreement,masi_distance,jaccard_distance,binary_distance
from nltk.metrics.agreement import AnnotationTask
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.metrics.pairwise import cosine_similarity
from scipy import spatial
import numpy as np

# cosine similarity is assuming full rank matrices, won't work for rank-deficient
# go through dimension reduction for matrices with all 0s column
def cosine_distance(vec_a,vec_b):
    distance = spatial.distance.cosine(vec_a,vec_b)
    return distance

# dropping columns with all 0s in the matrix
def dim_reduction(vector):
    vector = np.array(vector)
    idx_zero_cols = np.argwhere(np.all(vector[..., :] == 0, axis=0))
    res = np.delete(vector, idx_zero_cols, axis=1)
    return res

#sample pipeline
coder_dict = {"rater1":df_1, "rater2":df_2, "rater3":df_3}
answer_list = [{'sports'}, {'food', 'sports'}, {'music', 'education'}]
arr = dim_reduction(mlb.fit_transform(answer_list))
task_data = []
for pair in zip([list(r) for r in arr],coder_dict.keys()):
            answer, coder = pair
            annotation = coder, row["videoId"], tuple(answer)
            task_data.append(annotation)

cosine_task = AnnotationTask(data=task_data,distance = cosine_distance)
print(f"Fleiss's Kappa using Cosine distance: {cosine_task.multi_kappa()}")

