# Choosing estimator based on Precision-Recall Curve

I hope you will help with this one. Below you see my Precision-Recall curves after performing 10-fold Stratified Cross Validation on my dataset.

Note that my dataset is imbalanced that's why I'm checking the performance with this curve.

Given that I am interested in an estimator with high precision, which one should I choose in this case? Should the precision-recall curve be used as a guidance for choosing a classifier? Or the average precision is what I should just check?

The average precisions are:

• Random Forest: 0.807
• Logistic Regression: 0.601

• Just to be clear by Precision you refer to the ratio: $\frac{TP}{TP+FP}$, correct? Also can you please mention your sample size (approximately at least) and the class imbalance in your sample? This will help contextualise the question more. Finally: 1. When you stratify, how was the stratification done? 2. Can you please add calibration plots? (They are very informative in the case of imbalanced classification.) – usεr11852 Mar 17 '18 at 18:16
• Yes, this is what I mean for Precision. I have ~40k negative class and 5k~ positive (for which I want to optimise Precision). 1. The stratification was done using StratifiedShuffleSplit from sklearn. 2. Calibration plots added (very quickly produced, hopefully they are correct) – christinabo Mar 17 '18 at 19:21
• Thank you for these clarifications. Please see my answer below. – usεr11852 Mar 18 '18 at 17:05

As a side-comment: Try using RepeatedStratifiedKFold or RepeatedKFold instead of their one-pass variants. The repeated versions offer more stable insights as they decrease the variance of the metric we choose to examine.