10
$\begingroup$

I'm trying to make a detector of objects that occur very rarely (in images), planning to use a CNN binary classifier applied in a sliding/resized window. I've constructed balanced 1:1 positive-negative training and test sets (is it a right thing to do in such case btw?), and classifier is doing fine on a test set in terms of accuracy. Now I want to control recall/precision of my classifier so, for example, it will not wrongly label too much of a majority class occurrences.

Obvious (for me) solution is to use same logistic loss which is used now, but weight type I and type II errors differently by multiplying loss in one of the two cases on some constant, which can be tuned. Is it right?

P.S. On a second thought this is equivalent to weighting some training samples more than the others. Just adding more of one class will achieve the same I think.

$\endgroup$
  • $\begingroup$ did you ever resolve this? I have a similar objective. I would like to optimize for precision (type 1), and care less about type 2 errors, so have been considering what can be done with respect to loss function. $\endgroup$ – Jonathan Shore Jul 20 '17 at 15:06
6
$\begingroup$

Artificially constructing a balanced training set is debatable, quite controversial actually. If you do it, you should empirically verify that it really works better than leaving the training set unbalanced. Artificially balancing the test-set is almost never a good idea. The test-set should represent new data points as they come in without labels. You expect them to be unbalanced, so you need to know if your model can handle an unbalanced test-set. (If you don't expect new records to be unbalanced, why are all your existing records unbalanced?)

Regarding your performance metric, you will always get what you ask. If accuracy is not what you need foremost in an unbalanced set, because not only the classes but also the misclassification costs are unbalanced, then don't use it. If you had used accuracy as metric and done all your model selection and hyperparameter tuning by always taking the one with the best accuracy, you are optimizing for accuracy.

I take the minority class as the positive class, this is the conventional way of naming them. Thus precision and recall as discussed below are precision and recall of the minority class.

  • If the only important thing is to identify all the minority class records, you could take recall. You are thus accepting more false positives.
  • Optimizing only precision would be a very weird idea. You would be telling your classifier that it's not a problem to underdetect the minority class. The easiest way to have a high precision is to be overcautious in declaring the minority class.
  • If you need precision and recall, you could take F-measure. It is the harmonic mean between precision and recall and thus penalizes outcomes where both metrics diverge.
  • If you know the concrete misclassification costs in both directions (and the profits of correct classification if they are different per class), you can put all that in a loss function and optimize it.
$\endgroup$
3
$\begingroup$

You are making several assumptions. It is best to think of the ultimate goal in general terms, then formulate a strategy that meets that goal. For example do you really need forced-choice classification and is the signal:noise ratio large enough to support that (good examples: sound and image recognition)? Or is the signal:noise ratio low or you are interested in tendencies? For the latter, risk estimation is for you. The choice is key and dictates the predictive accuracy metric you choose. For more thoughts on all this see http://www.fharrell.com/2017/01/classification-vs-prediction.html and http://www.fharrell.com/2017/03/damage-caused-by-classification.html.

The majority of problems concern decision making, and optimum decisions come from risk estimation coupled with a loss/cost/utility function.

One of the best aspects of a risk (probability) estimation approach is that it handles gray zones where it would be a mistake to make a classification or decision without acquiring more data. And then there is the fact that probability estimation does not require (even does not allow) one to "balance" the outcomes by artificially manipulating the sample.

$\endgroup$
1
$\begingroup$

Regarding your question about whether reweighting training samples is equivalent to multiplying the loss in one of the two cases by a constant: yes, it is. One way to write the logistic regression loss function is $$\sum_{j=1}^J\log\left\{1+\exp\left[-f\left(x_j\right)\right]\right\}+\sum_{k=1}^K\log\left\{1+\exp\left[f\left(x_k\right)\right]\right\}$$ where $j$ and $k$ denote respective positive and negative instances, and $f(\cdot)$ is the logistic classifier built from features $x$. If you want to give more weight to your negative instances, for example, you might wish to modify your loss as
$$\sum_{j=1}^J\log\left\{1+\exp\left[-f\left(x_j\right)\right]\right\}+\sum_{k=1}^Kw\log\left\{1+\exp\left[f\left(x_k\right)\right]\right\}$$ for some $w>1$. This loss function is minimized by software implementations of weighted logistic regression, but you could also arrive at the same answer by upweighting your negative instances by a factor of $w$ and fitting a standard logistic regression (for example, if $w=2$, then you create 2 copies of each negative instance and fit). Some further details on this kind of approach here. And there is a general warning about what happens to parameter standard errors here, but this may not be such a concern if you're solely doing prediction.

$\endgroup$
  • $\begingroup$ But that would no longer be a maximum likelihood estimator - a statistical no-no $\endgroup$ – Frank Harrell Sep 8 '17 at 15:25
  • 1
    $\begingroup$ Agree, but I'm not convinced that it matters if statistical inference on the parameters in the logistic regression is not the desired goal (the OP's mention of using CNN is not ML-based either). Indeed, most/all inferential output from this weighted approach would best be ignored, but the model and resulting risk scores could still be applied to a validation set with desirable results, e.g. good discrimination/calibration. $\endgroup$ – Travis Gerke Sep 8 '17 at 15:54
  • 1
    $\begingroup$ No, you will disturb the calibration of the model and will get more noisy parameter estimates with the above approach. MLE exists for some very good reasons. $\endgroup$ – Frank Harrell Sep 8 '17 at 16:33
  • $\begingroup$ Consider the trivial case of being very concerned about misclassification one way, but not the other - i.e. zero loss for one of the directions. The best model for that loss would predict only the class of concern. Although it would be a horrible model, the objective is achieved. It is important to understand the objective and not put blind faith in a theoretical concept (MLE) without understanding it's purpose. As noted by TravisGerke , if the emphasis is on prediction rather than modeling, then his approach is quite useful. It's certainly better than downsampling the majority class. $\endgroup$ – Statseeker Apr 3 '18 at 23:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.