I'm working with an imbalanced binary classifier data set (3% positive) in sklearn. The cost of a false negative is extremely high so recall is much more important than precision.
To baseline my model I tuned:
- Random Forest depth, minsplitsize & numberFeatures using Average Precision (Area under Precision Recall curve)
- 5foldCV train on 70% of data, 30% are withheld for test.
- Balanced class weights are used to help with class imbalance.
On my test data I get good overall performance but too many false negatives, as a result I lowered my decision threshold until I had near 0 false negatives.
The other approach I was considering was to tune using the FBeta Statistic (Beta = 4) to favor recall over precision upfront with less adjustment to decision threshold thereafter.
1) Of the 2 approaches is one more correct for dealing with False Negatives?
2) In general is it bad to tune a binary classifier on a given metric and then move the decision threshold materially away from 0.5?
