2
$\begingroup$

I hope you will help with this one. Below you see my Precision-Recall curves after performing 10-fold Stratified Cross Validation on my dataset.

Note that my dataset is imbalanced that's why I'm checking the performance with this curve.

Given that I am interested in an estimator with high precision, which one should I choose in this case? Should the precision-recall curve be used as a guidance for choosing a classifier? Or the average precision is what I should just check?

The average precisions are:

  • Random Forest: 0.807
  • Gradient Boosting: 0.647
  • Logistic Regression: 0.601

precision recall curve

enter image description here

Improve this question
$\endgroup$
3
  • $\begingroup$ Just to be clear by Precision you refer to the ratio: $\frac{TP}{TP+FP}$, correct? Also can you please mention your sample size (approximately at least) and the class imbalance in your sample? This will help contextualise the question more. Finally: 1. When you stratify, how was the stratification done? 2. Can you please add calibration plots? (They are very informative in the case of imbalanced classification.) $\endgroup$
    – usεr11852
    Mar 17, 2018 at 18:16
  • $\begingroup$ Yes, this is what I mean for Precision. I have ~40k negative class and 5k~ positive (for which I want to optimise Precision). 1. The stratification was done using StratifiedShuffleSplit from sklearn. 2. Calibration plots added (very quickly produced, hopefully they are correct) $\endgroup$
    – christinabo
    Mar 17, 2018 at 19:21
  • $\begingroup$ Thank you for these clarifications. Please see my answer below. $\endgroup$
    – usεr11852
    Mar 18, 2018 at 17:05

1 Answer 1

Reset to default
1
$\begingroup$

Based on the PR-curves and Calibration plots shown the Gradient Boosting (GB) classifier seems to have the best performance.

With the exception of a small interval the GB classfier outperforms the alternative Random Forest (RF) classifier; both classifiers clearly outperform the Logistic Regression (LR) classifier used. In addition, the calibration curves for the GB classifier are the most sensible of the three. The RF classifier severely under-estimates certain probabilities of having a positive result (from the calibration plot it seems that for a mean predictive value of ~0.60 we expect about 85% positives). This is a clear sign that a classifier becomes overly stringent resulting to a higher precision that is mostly an artefact. Similarly the LR seems to over-estimate certain probabilities of having a positive result.

As this is a classification task, we must ultimately consider our utility function in terms of correct outcomes and choose the action that optimizes the expected utility. Precision is explicitly mentioned as a target but I would draw attention to the fact precision is based on a particular cut-off. Using a proper scoring rule like the logarithmic scoring rule or Brier score would be a more coherent alternative.

As a side-comment: Try using RepeatedStratifiedKFold or RepeatedKFold instead of their one-pass variants. The repeated versions offer more stable insights as they decrease the variance of the metric we choose to examine.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks a lot for these comments, there were very helpful! $\endgroup$
    – christinabo
    Mar 19, 2018 at 21:52
  • 1
    $\begingroup$ I am glad I could help! Καλή τύχη. $\endgroup$
    – usεr11852
    Mar 19, 2018 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.