Classifier with adjustable precision vs recall

Question

I am working on a binary classification problem where it is much more important to not have false positives; quite a lot of false negatives is ok. I have used a bunch of classifiers in sklearn for example, but I think none of them have the ability to adjust the precision-recall tradeoff explicitly (they do produce pretty good results but not adjustable).

What classifiers have adjustable precision/recall? Is there any way to influence the precision/recall tradeoff on standard classifiers, eg Random Forest or AdaBoost?

Answer 1

Almost all of scikit-learn's classifiers can give decision values (via decision_function or predict_proba).

Based on the decision values it is straightforward to compute precision-recall and/or ROC curves. scikit-learn provides those functions in its metrics submodule.

A minimal example, assuming you have data and labels with appropriate content:

import sklearn.svm
import sklearn.metrics
from matplotlib import pyplot as plt

clf = sklearn.svm.LinearSVC().fit(data, labels)
decision_values = clf.decision_function(data)

precision, recall, thresholds = sklearn.metrics.precision_recall_curve(labels, decision_values)

plt.plot(recall, precision)
plt.show()

Answer 2

I have just solved this for myself before bumping into this Q so I have decided to share my solution.

It uses the same approach that Marc Claesen has proposed but answers the actuall question on how to adjust the classifier to move higher on precision axis trading off the recall.

X_test is the data and y_test are the true labels. The classifier should be fitted already.

y_score = clf.decision_function(X_test)

prcsn,rcl,thrshld=precision_recall_curve(y_test,y_score)

min_prcsn=0.25 # here is your precision lower bound e.g. 25%
min_thrshld=min([thrshld[i] for i in range(len(thrshld)) if prcsn[i]>min_prcsn])

And this is how you'd use the newly learne d minimal threshold to adjust your prediction (that you would otherwise just get calling predict(X_test))

y_pred_adjusted=[1 if y_s>min_thrshld else 0 for y_s in y_score]

Would be great to hear your feedback on this adjustment recipe.

Answer 3

Here is function

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.predict_proba

which produces class probabilities. Then use various metrics to adjust threshold, P, which you use in decision by following pseudo-code

if p(y)>P, then y=1 else y=0 endif