0
$\begingroup$

I'm evaluating a series of binary classifiers and I care about how one performs better than the other for the same task, albeit under different settings. In my use case, I do not particularly care about the True Negative rate. I have been using binary/positive $F_{1}$-score to "focus" on the positive class in my evaluation, however, I have noticed where this can become a problem for comparison:

Take, for example, two models, $M_{A}$ and $M_{B}$, for comparison, with following ground-truth labels of shape (10,):

$Y = [0, 1, 0, 0, 1, 1, 1, 0, 1, 0]$

If $M_{A}$ predicts $[0, 1, 0, 0, 1, 0, 0, 0, 0, 0]$, I achieve an $F_{1}$-score of 0.5714 when $Y=1$

But if I have a model, $M_{B}$, which does not converge at all, and spams out 10 positive predictions, I get a score of 0.6667 (recurring), giving the appearance that $M_{B}$ outperforms $M_{A}$ despite not being able to discern between classes.

How can I emphasise the positive-class in that I don't care about True Negatives and perhaps False Negatives, but I do care about both True and False Positives?

I have looked into using Balanced Accuracy as a metric but I was a little afraid that, while this seems to capture label imbalance, it assumes that both labels are equally important. I have also looked into $F_{\beta}$-score in which I can assign a weight to either class, however, how do I "determine" this seemingly arbitrary weight?

$\endgroup$
2
  • 1
    $\begingroup$ If you don’t care about catching true negatives, why predict anything as a negative? Skip all of the fancy modeling. Your function is def apgsov_classifier(data): return(1). $\endgroup$
    – Dave
    Commented Jan 31, 2023 at 13:17
  • $\begingroup$ I do care about "catching" them, I don't care about reporting True Negatives when comparing the evaluation output from my models in which I'm primarily concerned with the positive class. $\endgroup$
    – apgsov
    Commented Jan 31, 2023 at 13:25

1 Answer 1

1
$\begingroup$

Do not use "hard" 0-1 classifications, but probabilistic classifications. Evaluate these using proper scoring rules. Separate the probabilistic prediction aspect from the subsequent decision, where you might indeed use thresholds, but only after carefully considering the costs of decisions versus outcomes. The F1 score (and all related KPIs, like accuracy, precision, sensitivity etc.) silently assume a very specific cost and conflate the two aspects. See the links in this thread.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.