2
$\begingroup$

I have trained a classifier I am experimenting on, using a highly imbalanced data set (284,807 transactions and out of them 370 are of class 1) and I get the following results.

          precision    recall  f1-score   support

       0       1.00      0.97      0.98     28432
       1       0.05      0.92      0.09        49

I use the predict_proba function from sklearn to get the probabilities matrix, and then I use the precision_recall_curve(test_labels, prob[:, 1]) to get the Precision and the Recall for my plot. But the resulting plot I get from pyplot.plot(recall, precision) comes out like this:

enter image description here

My question is it possible to have those results and still get a Precision Recall Curve looking like that and if yes how is it explained?

Thank you in advance for any answers.

$\endgroup$

1 Answer 1

1
$\begingroup$

PR curve plot displays metrics for every possible classification threshold. If you did not resample your data, the optimal threshold will be far from the default 0.5 used by predict() or classification_report() (which is somewhere in the lower right corner now).

However, PR curve isn't just a nice visualization. Its values can be used for the optimal threshold selection - for example, the best F1 metric:

probabilities = model.predict_proba(...)
precision, recall, thresholds = precision_recall_curve(test_labels, probabilities[:, 1])
# Vectorized operation:
f1_scores = 2 * recall * precision / (recall + precision)
best_f1 = np.max(f1_scores)
best_thresh = thresholds[np.argmax(f1_scores)]
best_predictions = (probabilities > best_thresh)
$\endgroup$
3
  • $\begingroup$ Thank you for your answer. I did not resample the data so the default threshold is 0.002 which is the probability of randomly selecting a sample from class 1. Since I can get the optimal threshold and f1-score, how can I use those to improve my model? Also what exactly do those optimal values mean? $\endgroup$ Jul 29, 2022 at 18:02
  • $\begingroup$ PR curve is a parametric one. Each point of it has a respective third dimension (threshold). Find a point with good precision/recall balance and you'll get a good threshold value to compare your predict_proba() result to. You model does not change at all. $\endgroup$
    – dx2-66
    Jul 29, 2022 at 19:47
  • $\begingroup$ Thank you very much now i understand. $\endgroup$ Aug 2, 2022 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.