I'm working with an imbalanced binary classifier data set (3% positive) in sklearn. The cost of a false negative is extremely high so recall is much more important than precision.

To baseline my model I tuned:

  • Random Forest depth, minsplitsize & numberFeatures using Average Precision (Area under Precision Recall curve)
  • 5foldCV train on 70% of data, 30% are withheld for test.
  • Balanced class weights are used to help with class imbalance.

On my test data I get good overall performance but too many false negatives, as a result I lowered my decision threshold until I had near 0 false negatives.

The other approach I was considering was to tune using the FBeta Statistic (Beta = 4) to favor recall over precision upfront with less adjustment to decision threshold thereafter.

1) Of the 2 approaches is one more correct for dealing with False Negatives?

2) In general is it bad to tune a binary classifier on a given metric and then move the decision threshold materially away from 0.5?

  • $\begingroup$ Accuracy is not being used - area under the precision recall curve is and/or the FBeta Statistic. $\endgroup$ – Nahyyz Jul 12 '18 at 11:12
  • $\begingroup$ The same logic applies. $\endgroup$ – Stephan Kolassa Jul 12 '18 at 11:47