I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed the ROC-AUC and the Precision-Recall-AUC (PR-AUC). Then the surprise came!
Method 1 is better than method 2 when I compare them using ROC-AUC.
Method 2 is better than method 1 when I compare them using PR-AUC.
Now I'm so confused! How to say which method is better? As far as I know from this paper that if the ROC-AUC is high, then PR-AUC is also high. So if the ROC curve of method-1 dominates, so should method-1's PR curve. Is my understanding incorrect? Or am I missing something? Because I'm really going crazy!