Choosing estimator based on Precision-Recall Curve I hope you will help with this one. Below you see my Precision-Recall curves after performing 10-fold Stratified Cross Validation on my dataset.
Note that my dataset is imbalanced that's why I'm checking the performance with this curve. 
Given that I am interested in an estimator with high precision, which one should I choose in this case? Should the precision-recall curve be used as a guidance for choosing a classifier? Or the average precision is what I should just check?
The average precisions are:


*

*Random Forest: 0.807

*Gradient Boosting: 0.647

*Logistic Regression: 0.601




 A: Based on the PR-curves and Calibration plots shown the Gradient Boosting (GB) classifier seems to have the best performance. 
With the exception of a small interval the GB classfier outperforms the alternative Random Forest (RF) classifier; both classifiers clearly outperform the Logistic Regression (LR) classifier used. In addition, the calibration curves for the GB classifier are the most sensible of the three. The RF classifier severely under-estimates certain probabilities of having a positive result (from the calibration plot it seems that for a mean predictive value of ~0.60 we expect about 85% positives). This is a clear sign that a classifier becomes overly stringent resulting to a higher precision that is mostly an artefact. Similarly the LR seems to over-estimate certain probabilities of having a positive result. 
As this is a classification task, we must ultimately consider our utility function in terms of correct outcomes and choose the action that optimizes the expected utility. Precision is explicitly mentioned as a target but I would draw attention to the fact precision is based on a particular cut-off. Using a proper scoring rule like the logarithmic scoring rule or Brier score would be a more coherent alternative.
As a side-comment: Try using RepeatedStratifiedKFold or RepeatedKFold instead of their one-pass variants. The repeated versions offer more stable insights as they decrease the variance of the metric we choose to examine.
