I have a data set with around 250 data points. I previously automatically labeled the data using an established classifier. Every data point is assigned with one of three possible labels. I then conducted a user study where ten users labeled 25 randomly chosen data points (from the 250, none used twice).

I now have the labels of the classifier and the user annotated labels. How do i measure the agreement of both the users and the classifier? I don't know if i just can use Precision / Recall and F1-score as metrics to evaluate the agreement of the users and the classifier?

  • $\begingroup$ You might want to explore Cohen's kappa. I do not think precision and recall work well with three-valued variables but I might be wrong. $\endgroup$ – mdewey Mar 8 '17 at 10:02
  • 1
    $\begingroup$ I thought about Cohen's kappa but i am not sure if that is possible with the 10 different users. I would have to use the total of 10x25 user annotations as one set and the classifier as another? $\endgroup$ – Symn Mar 8 '17 at 10:33
  • 1
    $\begingroup$ There is a multi-rater version en.wikipedia.org/wiki/Fleiss%27_kappa $\endgroup$ – mdewey Mar 8 '17 at 10:43
  • $\begingroup$ +1 for mdewey Check out Joe Fleiss's text on this subject. $\endgroup$ – Michael Chernick Mar 8 '17 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.