3
$\begingroup$

I am trying to understand the interpretation of these metrics in a multiclass scenario: ROC-AUC and MAUC. Scikit-learn provides an implementation for ROC-AUC score, which can be used for both binary and multiclass problems.

However, some studies such as 2, 3, 4 and 5 suggest averaging class-wise AUC in multiclass.

My experiments with this metrics yield different results. Do they then evaluate to different quantities?

For clarity, I used 4 implementation as well as sklearn's ROC-AUC. I the case of sklearn's, I set the hyperparameters as:

roc_auc = metrics.roc_auc_score(y_test, ypred, average='weighted', 
    multi_class='ovo',labels=labels)

With Random Forest classifier, we obtained:

ROC-AUC:  0.58 # sklearn's roc-auc-score
MAUC:     0.69

This is more than a 10% difference so the two values are not close at all.

EDIT

References:

1 sklearn's roc-auc score: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html

2 Tanha, J., Abdi, Y., Samadi, N., Razzaghi, N., Asadpour, M.: Boosting methods for multi-class imbalanced data classification: an experimental review. Journal of Big Data 7(1), 1–47 (2020)

3 Wang, R., Tang, K.: An empirical study of MAUC in multi-class problems with uncertain cost matrices. CoRR abs/1209.1800 (2012), http://arxiv.org/abs/1209.1800

4 https://gist.github.com/stulacy/672114792371dc13b247

5 https://github.com/pritomsaha/Multiclass_AUC/blob/master/multiclass_auc.ipynb

$\endgroup$
2
  • 2
    $\begingroup$ Could you please include full citations for those references? If you have the titles, it’s usually pretty easy to find full citations on Google Scholar. That way, if the links rot, we still have the references. $\endgroup$
    – Dave
    Apr 10 at 15:30
  • 1
    $\begingroup$ You may also want to reenumerate the references in the paragraph. $\endgroup$
    – gunes
    Apr 10 at 16:07

2 Answers 2

1
$\begingroup$

The sklearn implementation offers different options for multi_class and average which explains the main difference: the Hand-Till paper and the implementations you linked use a one-vs-one approach as you do in your sklearn call, but also uses a macro-average compared to your weighted average approach.

There's another issue that prevents the scores from agreeing completely, though it will be more minor than the averaging issue. The gist and github implementations both sort the samples by probability, but ties are left up to the numpy sorting. In sklearn however, a tie of probabilities is handled by having a sloped line in the ROC curve, which also affects the area computation. Tweaking the toy example in [5] (since gunes pointed out the difference there, even though the classes are balanced and so the averaging plays no role) to have no ties in probability scores yields equal scores.

$\endgroup$
1
  • $\begingroup$ thank you for this detail answer. $\endgroup$
    – arilwan
    Apr 11 at 9:55
2
$\begingroup$

Reference [5] seems to use an approximation in the function a_value but that doesn't seem to be the way the AUC is calculated in sklearn. It uses the method _average_multiclass_ovo_score, whose binary_metric is given as _binary_roc_auc_score. This method calls the roc_curve and general purpose auc method. I don't think this implements the logic embedded in the fourth reference.

Even the toy example given in [5] does not give the same value as sklearn.

$\endgroup$
5
  • $\begingroup$ sklearn's ovo "Computes the average AUC of all possible pairwise combinations of classes" as far the documentation. Maybe you mean sklearn's ovr? $\endgroup$
    – arilwan
    Apr 10 at 15:26
  • $\begingroup$ You're right. Will look into sklearn's implementation. $\endgroup$
    – gunes
    Apr 10 at 15:48
  • $\begingroup$ Does this suggest the tow metric measures are different? $\endgroup$
    – arilwan
    Apr 10 at 16:35
  • 2
    $\begingroup$ Yes, it appears the sklearn does not use the approximated formula, which is equation 3 in this paper: link.springer.com/content/pdf/10.1023/A:1010920819831.pdf $\endgroup$
    – gunes
    Apr 10 at 16:40
  • 1
    $\begingroup$ I think Hand and Till use "estimate" to mean using a test sample to estimate the true/asymptotic AUC. The formula they give is equivalent to the area under the ROC curve, except that ties need to be handled (they call this out in the last line of page 173, but I don't see any further mention of how to adjust the procedure when there are ties?). $\endgroup$ Apr 11 at 2:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.