1
$\begingroup$

I currently have a model that's tasked with multilabel classification with an extra "no answer" option for negative predictions. The metrics that I'm using are precision, recall, and the F1 score. The be more specific, it's a relation classification task where the model is provided text and pairs of entities and is tasked with classifying the relation for each pair.

To roughly outline the evaluation pipeline I have:

  1. Extract all positive predictions from all of the predictions (i.e., samples where the model didn't predict "no answer"). The total number of positive predictions is the denominator for our precision (i.e., TP + FP).
  2. Among these positive predictions, compare with the ground truth dev/test set and see how many (sample, label) instances overlap. The number of overlapping instances will be our true positive count.
  3. The total number of samples in the dev/test set will be the denominator for recall (i.e., TP + FN).
  4. Calculate precision, recall, and F1 as usual.

First off there are a ton of negative predictions. Among the (roughly) 100,000 samples, I think around 80-90% are labeled as "no answer." I suppose that's not exactly outrageous, but what's also concerning me is that my precision is 1.0 and recall is 0.53, leading to an overall F1 score of around 0.70, which I've personally never seen before.

How should I interpret whether this perfect precision/mediocre recall scenario is acceptable or not? Any thoughts are appreciated, thanks.

$\endgroup$

1 Answer 1

0
$\begingroup$

my precision is 1.0 and recall is 0.53

If you think it in terms of binary classification (positives and negatives), this means that of all the actual positives, your model predicted only 53% of them correctly as True positives (recall) (bad thing), at the same time, those 53% that your model predicted as True positives were in fact positives (precision) (good thing).

In general sense, this doesn't seem a good thing.

You should look at class-wise precision-recall values to better analyze your model's performance w.r.t each class.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.