I have a data set with around 250 data points. I previously automatically labeled the data using an established classifier. Every data point is assigned with one of three possible labels. I then conducted a user study where ten users labeled 25 randomly chosen data points (from the 250, none used twice).
I now have the labels of the classifier and the user annotated labels. How do i measure the agreement of both the users and the classifier? I don't know if i just can use Precision / Recall and F1-score as metrics to evaluate the agreement of the users and the classifier?