3
$\begingroup$

I currently have a problem where I am trying to classify medical abstracts where some are relevant and some aren't. I have tried an SVM, Multinomial Naive Bayes and Random Forest, and found the MNB works best for the task at hand. The only issue is, my recall for the "relevant" class is slightly lower than it should be. For the datasets I've tested on, my recall is between 0.6-0.85.

Ideally I want the recall for this class to be 0.95+, even if it comes with a decrease in overall classification rate. I would like to minimize the number of False Negatives as much as possible, even if that means the number of False Positives shoot up.

I'm using scikit-learn to implement my models. I tried running a grid search with different alphas and my own scoring function (recall for the class I want) but it just returned the same parameters I was using before.

Any help would be greatly appreciated.

Improve this question
$\endgroup$
1
  • $\begingroup$ you could also tweak the class_weights (or sample_weight for your classifier) in favor of the positive class, as suggested in this answer $\endgroup$
    – mork
    Commented Apr 2, 2019 at 15:36

1 Answer 1

Reset to default
3
$\begingroup$

You don't need to change anything in the model itself you just need to change the place where you put your decision threshold.

For example, if the naive-bayes returns the probability of the positive class then by default scikit-learn will put the decision threshold at 0.5, by getting the probabilities out of the classifier instead of the predicted label you can shift this decision threshold to better suit your particular needs.

This may come at the cost of some overall accuracy, the best way to decide the optimal threshold of your particular case is to look at the ROC curve, (here is an implementation using skit-learn) where you plot the True Positive Rate (TPR) against the False Negative Rate (FPR) as you vary the decision threshold (in the case of naive-bayes from 0 to 1). Just pick a point that gets you a recall (same as TPR) above .95 and make sure your not compromising too much FPR (if so look at the ROC curves for other classifiers).

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you very much for this answer Miguel. The probabilities on the MNB were quite close together and it was difficult to find a threshold that offered any substantial improvement, however using the outputs from the svm decision function, I was able to achieve 0.95+ recall on all my holdout datasets with an adequate threshold. $\endgroup$
    – user168363
    Commented Jul 18, 2017 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.