I was wondering whether there is a metric that can be used in order to compute the agreement, and therefore something like an upper bound for classifiers, among expert-labelled data.
Assume there is a multi-label problem where $N$ documents have to be tagged using tags from a set of tags (e.g. car, house, animal) and three experts.
Document 1 Document 2 Document 3 Document 4 Expert 1: [car, house] [animal] [car, animal]  Expert 2: [car] [animal] [car, animal]  Expert 3: [car, house] [animal, house] [car, animal] [animal]
Are there ways to compute an "agreement score" and ultimately determine an upper bound for an artificial classifier?