# Evaluating binary classifiers with emphasis on positive-class

I'm evaluating a series of binary classifiers and I care about how one performs better than the other for the same task, albeit under different settings. In my use case, I do not particularly care about the True Negative rate. I have been using binary/positive $$F_{1}$$-score to "focus" on the positive class in my evaluation, however, I have noticed where this can become a problem for comparison:

Take, for example, two models, $$M_{A}$$ and $$M_{B}$$, for comparison, with following ground-truth labels of shape (10,):

$$Y = [0, 1, 0, 0, 1, 1, 1, 0, 1, 0]$$

If $$M_{A}$$ predicts $$[0, 1, 0, 0, 1, 0, 0, 0, 0, 0]$$, I achieve an $$F_{1}$$-score of 0.5714 when $$Y=1$$

But if I have a model, $$M_{B}$$, which does not converge at all, and spams out 10 positive predictions, I get a score of 0.6667 (recurring), giving the appearance that $$M_{B}$$ outperforms $$M_{A}$$ despite not being able to discern between classes.

How can I emphasise the positive-class in that I don't care about True Negatives and perhaps False Negatives, but I do care about both True and False Positives?

I have looked into using Balanced Accuracy as a metric but I was a little afraid that, while this seems to capture label imbalance, it assumes that both labels are equally important. I have also looked into $$F_{\beta}$$-score in which I can assign a weight to either class, however, how do I "determine" this seemingly arbitrary weight?

• If you don’t care about catching true negatives, why predict anything as a negative? Skip all of the fancy modeling. Your function is def apgsov_classifier(data): return(1).
– Dave
Jan 31 at 13:17
• I do care about "catching" them, I don't care about reporting True Negatives when comparing the evaluation output from my models in which I'm primarily concerned with the positive class. Jan 31 at 13:25