0
$\begingroup$

The classifier I'm using has 3 possible label outputs - POSITIVE, NEGATIVE or UNKNOWN. For training data, the labels are only POSITIVE and NEGATIVE.

What is the best way to handle evaluating the classifier output? I want to preserve the UNKNOWN label in general since I don't want low-confidence labels, but I want to also minimize the amount of UNKNOWNs while preserving precision/recall.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ How do you arrive at an "unknown" classification? $\endgroup$
    – Dave
    Commented Mar 22, 2023 at 15:55
  • $\begingroup$ The data is clustered and then clusters are labeled from rules. If the cluster doesn't match any rules, it's labeled UNKNOWN. $\endgroup$
    – filaments
    Commented Mar 22, 2023 at 16:01

1 Answer 1

Reset to default
1
$\begingroup$

Use a probabilistic classifier. Anything that gets a low predicted probability of belonging to the target class gets labeled NEGATIVE. Anything with a high predicted probability gets labeled POSITIVE. Anything in between is UNKNOWN. Adjust the two thresholds involved as necessary to optimize your KPIs.

Note that Precision and Recall suffer from the exact same issues as Accuracy: Why is accuracy not the best measure for assessing classification models?

You may want to take a look at this answer about classification thresholds.

Improve this answer
$\endgroup$
2
  • $\begingroup$ It's the optimization part I'm wondering about. Regardless of how I arrive at the UNKNOWN label, I'm not sure the best way to evaluate performance to both minimize the amount of unknowns and maximize some other evaluation metric. $\endgroup$
    – filaments
    Commented Mar 22, 2023 at 16:04
  • 1
    $\begingroup$ If you want to optimize two different metrics, you will need to look into multicriteria optimization. The easiest approach is to use a weighted mean or sum of the two criteria. But whatever you do, you will need to think about how to trade off one against the other. $\endgroup$
    – Stephan Kolassa
    Commented Mar 22, 2023 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.