What loss function should one use to get a high precision or high recall binary classifier?

Question

I'm trying to make a detector of objects that occur very rarely (in images), planning to use a CNN binary classifier applied in a sliding/resized window. I've constructed balanced 1:1 positive-negative training and test sets (is it a right thing to do in such case btw?), and classifier is doing fine on a test set in terms of accuracy. Now I want to control recall/precision of my classifier so, for example, it will not wrongly label too much of a majority class occurrences.

Obvious (for me) solution is to use same logistic loss which is used now, but weight type I and type II errors differently by multiplying loss in one of the two cases on some constant, which can be tuned. Is it right?

P.S. On a second thought this is equivalent to weighting some training samples more than the others. Just adding more of one class will achieve the same I think.

Answer 1

Artificially constructing a balanced training set is debatable, quite controversial actually. If you do it, you should empirically verify that it really works better than leaving the training set unbalanced. Artificially balancing the test-set is almost never a good idea. The test-set should represent new data points as they come in without labels. You expect them to be unbalanced, so you need to know if your model can handle an unbalanced test-set. (If you don't expect new records to be unbalanced, why are all your existing records unbalanced?)

Regarding your performance metric, you will always get what you ask. If accuracy is not what you need foremost in an unbalanced set, because not only the classes but also the misclassification costs are unbalanced, then don't use it. If you had used accuracy as metric and done all your model selection and hyperparameter tuning by always taking the one with the best accuracy, you are optimizing for accuracy.

I take the minority class as the positive class, this is the conventional way of naming them. Thus precision and recall as discussed below are precision and recall of the minority class.

• If the only important thing is to identify all the minority class records, you could take recall. You are thus accepting more false positives.
• Optimizing only precision would be a very weird idea. You would be telling your classifier that it's not a problem to underdetect the minority class. The easiest way to have a high precision is to be overcautious in declaring the minority class.
• If you need precision and recall, you could take F-measure. It is the harmonic mean between precision and recall and thus penalizes outcomes where both metrics diverge.
• If you know the concrete misclassification costs in both directions (and the profits of correct classification if they are different per class), you can put all that in a loss function and optimize it.

Answer 2

Not too long after you asked this question, there was an interesting research paper entitled Scalable Learning of Non-Decomposable Objectives that I stumbled across from a StackOverflow question that finds ways to build several interesting loss functions:

• Precision at fixed recall
• Recall at fixed precision
• AUCROC maximization

There was an implementation for TF 1.x over here. Unfortunately it does not appear to have garnered much attention so it is not being actively maintained; however, I think this is a quite valuable approach when trying to build real-world binary classifiers.

Answer 3

You are making several assumptions. It is best to think of the ultimate goal in general terms, then formulate a strategy that meets that goal. For example do you really need forced-choice classification and is the signal:noise ratio large enough to support that (good examples: sound and image recognition)? Or is the signal:noise ratio low or you are interested in tendencies? For the latter, risk estimation is for you. The choice is key and dictates the predictive accuracy metric you choose. For more thoughts on all this see http://www.fharrell.com/2017/01/classification-vs-prediction.html and http://www.fharrell.com/2017/03/damage-caused-by-classification.html.

The majority of problems concern decision making, and optimum decisions come from risk estimation coupled with a loss/cost/utility function.

One of the best aspects of a risk (probability) estimation approach is that it handles gray zones where it would be a mistake to make a classification or decision without acquiring more data. And then there is the fact that probability estimation does not require (even does not allow) one to "balance" the outcomes by artificially manipulating the sample.

Answer 4

Regarding your question about whether reweighting training samples is equivalent to multiplying the loss in one of the two cases by a constant: yes, it is. One way to write the logistic regression loss function is $$\sum_{j=1}^J\log\left\{1+\exp\left[-f\left(x_j\right)\right]\right\}+\sum_{k=1}^K\log\left\{1+\exp\left[f\left(x_k\right)\right]\right\}$$ where $j$ and $k$ denote respective positive and negative instances, and $f(\cdot)$ is the logistic classifier built from features $x$. If you want to give more weight to your negative instances, for example, you might wish to modify your loss as
$$\sum_{j=1}^J\log\left\{1+\exp\left[-f\left(x_j\right)\right]\right\}+\sum_{k=1}^Kw\log\left\{1+\exp\left[f\left(x_k\right)\right]\right\}$$ for some $w>1$. This loss function is minimized by software implementations of weighted logistic regression, but you could also arrive at the same answer by upweighting your negative instances by a factor of $w$ and fitting a standard logistic regression (for example, if $w=2$, then you create 2 copies of each negative instance and fit). Some further details on this kind of approach here. And there is a general warning about what happens to parameter standard errors here, but this may not be such a concern if you're solely doing prediction.