Evaluating and combining methods based on ROC and PR curves I am evaluating and combining a few binary classification models. I am using the ROC and PR curves to evaluate their performance. The problem I am having is that as I try to improve the method, I am improving the AUC-ROC but the PR curve suffers. For example:
 
As an aside, I am actually adding a weak learner to Method 1 to arrive at Method 2, and then adding another weak  learner or two to arrive at Method 3. When I was only evaluating AUC-ROC, it looked fine, but when I saw the PR curve, it seems I have been degrading the performance. Now it seems that the weak learners are doing better at points lower in the ranked list. But this is only for one dataset training/test split. What would be a principled way to investigate what is going on and come up with a way to use the weak learners so as to improve both ROC and PR curves?
Update:
To visualize this, I am showing the weak learner that I am adding to Model 1 to arrive at Model 2 here:
 
 A: I will state a few things about the ROC / PR spaces that are surely evident for you but that I prefer to make clear.


*

*The ROC space is on the $x$-axis one minus the specificity : $1-Sp$, and on the $y$-axis the sensitivity : $Se$.

*The PR space is on the $x$-axis the recall, which is an other name of the sensitivity : $Re = Se$, and on the $y$-axis the precision, which is an other name of the Positive Predictive Value : $Pr = PPV$ ;

*If $p$ is the probability of being in the "positive class", we have
$$Pr = PPV = {Se\cdot p \over (1-Sp)\cdot(1-p) + Se \cdot p}.$$ 
The "horizontal slices" in the ROC space correspond to "vertical slices" of PR space. From the above equality, it is easy to see that when in the ROC space a curve (eg the red curve of your first graph) is on the left of a second one (the green curve), in the PR space the corresponding (red) curve is above the (green) curve.
This is the case in your second graph, except for Recall values $< 0.1$. The corresponding part of the ROC curves in your first graph is for Se $< 0.1$ which is "glued" to the $y$-axis, and you can’t see anything. Here the advantage of the PR space is that it helps visualizing this area.
So I don’t see contradiction in these results : method 3 is indeed better than the two others, except for Sensitivity / Recall values $< 0.1$, which correspond to very high Specificity values. 
The morality is that the way you improve your classifier slightly degrades its performances when you demand it to have a very high Specificity.
These are quite trivial reflexions, but who knows, this can help?
A: Deviance (or -2 log likelihood) is the most statistically sensitive measure.  I would use that to compare models.
A: For imbalanced classes using AUC as a measure of classifier performance, rather than (0,1)-loss can be misleading. See for example Xue and Titterington "Do unbalanced data have a
negative effect on LDA?". For two-class classification the (0,1)-loss is usually the loss of real interest, so you may find that working directly with that loss, rather than AUC, is more informative.
A: I eventually resorted to using logistic regression (and similar models like adaptive splines) etc. to combine the scores. I think the idea is that of stacking and has been used before, e.g., here and here.
