Calculating Area Under The Precision Recall Curve with multiclass

Question

I'm trying to calculate AUPR and when I was doing it on Datasets which were binary in terms of their classes, I used average_precision_score from sklearn and this has approximately solved my problem. However, when I tried to calculate average precision score on a multiclass dataset then its not supported according to sklearn.

Assuming I have to do this manually instead of using some sklearn metric, I'm trying to understand how can I calculate this and I would love some help.

Thanks!

Answer 1

You can calculate precision per class then take the average.

from sklearn.metrics import precision_recall_curve
from sklearn.metrics import average_precision_score

# For each class
precision = dict()
recall = dict()
average_precision = dict()
for i in range(n_classes):
    precision[i], recall[i], _ = precision_recall_curve(Y_test[:, i],
                                                        y_score[:, i])
    average_precision[i] = average_precision_score(Y_test[:, i], y_score[:, i])

# A "micro-average": quantifying score on all classes jointly
precision["micro"], recall["micro"], _ = precision_recall_curve(Y_test.ravel(),
    y_score.ravel())
average_precision["micro"] = average_precision_score(Y_test, y_score,
                                                     average="micro")
print('Average precision score, micro-averaged over all classes: {0:0.2f}'
      .format(average_precision["micro"]))

Check these sklearn Precision-Recall examaples.

Answer 2

As a side note, there is a multi-class implementation of the average precision in the torchmetrics module that also supports different averaging policies. Note that you would need to convert your numpy ndarrays with ground-truth labels and predictions into torch Tensors via torch.from_numpy() to use this implementation.