I am evaluating and combining a few binary classification models. I am using the ROC and PR curves to evaluate their performance. The problem I am having is that as I try to improve the method, I am improving the AUC-ROC but the PR curve suffers. For example:
As an aside, I am actually adding a weak learner to Method 1 to arrive at Method 2, and then adding another weak learner or two to arrive at Method 3. When I was only evaluating AUC-ROC, it looked fine, but when I saw the PR curve, it seems I have been degrading the performance. Now it seems that the weak learners are doing better at points lower in the ranked list. But this is only for one dataset training/test split. What would be a principled way to investigate what is going on and come up with a way to use the weak learners so as to improve both ROC and PR curves?
To visualize this, I am showing the weak learner that I am adding to Model 1 to arrive at Model 2 here: