Measure agreement among experts in multi-label classification task

I was wondering whether there is a metric that can be used in order to compute the agreement, and therefore something like an upper bound for classifiers, among expert-labelled data.

Assume there is a multi-label problem where $N$ documents have to be tagged using tags from a set of tags (e.g. car, house, animal) and three experts.

           Document 1    Document 2       Document 3     Document 4
Expert 1:  [car, house]  [animal]         [car, animal]  []
Expert 2:  [car]         [animal]         [car, animal]  []
Expert 3:  [car, house]  [animal, house]  [car, animal]  [animal]


Are there ways to compute an "agreement score" and ultimately determine an upper bound for an artificial classifier?

• have you considered inter rater reliability statistics? – Dimitrios Zacharatos Jun 26 '18 at 14:10
• Well, I've been reading about the kappa score but this seems to be limited to just two "experts" and does not seem to work for multi-label problems. I could work around that and just apply it for each class separately but I it seems I can only compare two outcomes at once. – displayname Jun 26 '18 at 14:26
• Fleis's Kappa solves the issue regarding the number of experts. – Dimitrios Zacharatos Jun 26 '18 at 14:56
• @DimitriosZacharatos Yes indeed, I just found this paper which appears to give a very nice overview of different methods and metrics. :) – displayname Jun 26 '18 at 15:05