Based on the PR-curves and Calibration plots shown the Gradient Boosting (GB) classifier seems to have the best performance.
With the exception of a small interval the GB classfier outperforms the alternative Random Forest (RF) classifier; both classifiers clearly outperform the Logistic Regression (LR) classifier used. In addition, the calibration curves for the GB classifier are the most sensible of the three. The RF classifier severely under-estimates certain probabilities of having a positive result (from the calibration plot it seems that for a mean predictive value of ~0.60 we expect about 85% positives). This is a clear sign that a classifier becomes overly stringent resulting to a higher precision that is mostly an artefact. Similarly the LR seems to over-estimate certain probabilities of having a positive result.
As this is a classification task, we must ultimately consider our utility function in terms of correct outcomes and choose the action that optimizes the expected utility. Precision is explicitly mentioned as a target but I would draw attention to the fact precision is based on a particular cut-off. Using a proper scoring rule like the logarithmic scoring rule or Brier score would be a more coherent alternative.
As a side-comment: Try using RepeatedStratifiedKFold
or RepeatedKFold
instead of their one-pass variants. The repeated versions offer more stable insights as they decrease the variance of the metric we choose to examine.