What does it mean if the ROC AUC is high and the Average Precision is low?

Question

I have a model that produces a high ROC AUC (0.90), but at the same time a low average precision (0.30). From what I've found, I think it might have to do something with imbalanced data (which the dataset is). However, I cannot see how this imbalance results in this significant difference. Reading Davis' paper has not gotten me to an answer either.

Answer 1

I came up with the same problem recently and found some help in a few posts, which are referenced at the end of this answer.

As usual I will use the abbreviations commonly used in the Confusion Matrix context: TP (True Positives), FP (False Positives), TN (True Negatives), FN (False Negatives). I will also consider the positive class to be the minority class, while the negative class is the majority class.

First, you should notice that ROC AUC and Precision-Recall AUC are ranking metrics [1]. This means that they measure how well your probabilities (or scores) can order your data. ROC and Precision-Recall curves are related to ordering, because of the variation of the threshold that is used so as to build the curves.

The difference between these metrics is how the ordering quality is quantified [2]. ROC analysis uses True Positive Rate (TPR or Recall) and False Positive Rate (FPR). Precision-Recall analysis, on the other hand, exchanges FPR for Precision. Then, while ROC uses all the cells (TP, FP, TN, FN) of the Confusion Matrix, Precision-Recall disregards the True Negatives, which have a high impact on an imbalanced problem, since almost all your data is of the negative class. Therefore, Precision-Recall gives more weight to the minority class (the positive class) than the ROC. This is why the Precision-Recall AUC is more suitable for heavily imbalanced problems.

The more intuitive meaning of having a high ROC AUC, but a low Precision-Recall AUC is that your model can order very well your data (almost of of them belong to the same class anyway), but high scores do not correlate well with being positive class. You are not very confident about your high scores, but is very confident about the low scores.

[1] https://machinelearningmastery.com/tour-of-evaluation-metrics-for-imbalanced-classification/

[2] https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-imbalanced-classification/

Answer 2

The floor of ROC AUC is 0.5, while the floor of AUCPR is the positive rate in your data. If your positive rate is low enough, an AUCPR of 0.3 is outstanding.