Evaluating binary classifiers with emphasis on positive-class

Question

I'm evaluating a series of binary classifiers and I care about how one performs better than the other for the same task, albeit under different settings. In my use case, I do not particularly care about the True Negative rate. I have been using binary/positive $F_{1}$-score to "focus" on the positive class in my evaluation, however, I have noticed where this can become a problem for comparison:

Take, for example, two models, $M_{A}$ and $M_{B}$, for comparison, with following ground-truth labels of shape (10,):

$Y = [0, 1, 0, 0, 1, 1, 1, 0, 1, 0]$

If $M_{A}$ predicts $[0, 1, 0, 0, 1, 0, 0, 0, 0, 0]$, I achieve an $F_{1}$-score of 0.5714 when $Y=1$

But if I have a model, $M_{B}$, which does not converge at all, and spams out 10 positive predictions, I get a score of 0.6667 (recurring), giving the appearance that $M_{B}$ outperforms $M_{A}$ despite not being able to discern between classes.

How can I emphasise the positive-class in that I don't care about True Negatives and perhaps False Negatives, but I do care about both True and False Positives?

I have looked into using Balanced Accuracy as a metric but I was a little afraid that, while this seems to capture label imbalance, it assumes that both labels are equally important. I have also looked into $F_{\beta}$-score in which I can assign a weight to either class, however, how do I "determine" this seemingly arbitrary weight?

Answer 1

Do not use "hard" 0-1 classifications, but probabilistic classifications. Evaluate these using proper scoring rules. Separate the probabilistic prediction aspect from the subsequent decision, where you might indeed use thresholds, but only after carefully considering the costs of decisions versus outcomes. The F1 score (and all related KPIs, like accuracy, precision, sensitivity etc.) silently assume a very specific cost and conflate the two aspects. See the links in this thread.