I was wondering whether there is a metric that can be used in order to compute the agreement, and therefore something like an upper bound for classifiers, among expert-labelled data.

Assume there is a multi-label problem where $N$ documents have to be tagged using tags from a set of tags (e.g. car, house, animal) and three experts.

           Document 1    Document 2       Document 3     Document 4
Expert 1:  [car, house]  [animal]         [car, animal]  []
Expert 2:  [car]         [animal]         [car, animal]  []
Expert 3:  [car, house]  [animal, house]  [car, animal]  [animal]

Are there ways to compute an "agreement score" and ultimately determine an upper bound for an artificial classifier?

  • $\begingroup$ have you considered inter rater reliability statistics? $\endgroup$ – Dimitrios Zacharatos Jun 26 '18 at 14:10
  • $\begingroup$ Well, I've been reading about the kappa score but this seems to be limited to just two "experts" and does not seem to work for multi-label problems. I could work around that and just apply it for each class separately but I it seems I can only compare two outcomes at once. $\endgroup$ – displayname Jun 26 '18 at 14:26
  • $\begingroup$ Fleis's Kappa solves the issue regarding the number of experts. $\endgroup$ – Dimitrios Zacharatos Jun 26 '18 at 14:56
  • $\begingroup$ @DimitriosZacharatos Yes indeed, I just found this paper which appears to give a very nice overview of different methods and metrics. :) $\endgroup$ – displayname Jun 26 '18 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.