I know that given an ordered set of binary labels, and equally-sized ordered set of scalar predictions, we can quantify how cleanly the predictions separate the labels into clean buckets of 0's and 1's using metrics like ROC-AUC or PR-AUC. What I'm wondering is, does there exist a natural extension of these ideas to multiple simultaneous classifications tasks, where we have n-many sets of binary labels, and n-many corresponding sets of scalar predictions, and we want an aggregate measure of how well the predictions separate their corresponding label sets' 0's and 1's?

Obviously I could just compute the AUC on every pair of label/prediction sets separately, and take the average, but that doesn't feel very clean. And it doesn't make sense to simply aggregate all of the label sets, and all of the prediction sets, and compute the AUC on the combined set, since each subset may have a different "cutoff" (see below example).

I think part of the problem is that this is really more of a coarse-grained information-retrieval/ranking task, as opposed to classification task which is what these metrics were designed for.

Illustration of why simply concatenating the labels and predictions doesn't work. The predictions separate the labels perfectly on each of the individual tasks, but performs poorly on the "aggregated" task: $$ \text{labels}_1 = \{0,0,1,1\},\ \text{preds}_1 = \{0.1, 0.2, 0.3, 0.4\}\\ \text{labels}_2 = \{0,0,1,1\},\ \text{preds}_2 = \{0.6, 0.7, 0.8, 0.9\}\\ \text{ROC_AUC}\big(\text{labels}_1,\ \text{preds}_1\big) = 1.0\\ \text{ROC_AUC}\big(\text{labels}_2,\ \text{preds}_2\big) = 1.0\\ \text{ROC_AUC}\big(\text{labels}_1\cup\text{labels}_2,\ \text{preds}_1\cup\text{preds}_2\big) = 0.75 $$

  • 1
    $\begingroup$ Have you looked into discussions on micro and macro metrics? I am not sure how to answer this question, but I think that might be the right direction. Or it might point you to some new papers on this topic. datascience.stackexchange.com/questions/15989/… $\endgroup$ – user0 Aug 31 '19 at 23:50
  • $\begingroup$ I see, so the difference between macro and micro averaging is basically just the difference between taking a naive unweighted average of the per-class metrics, and taking a #samples-weighted average of the per-class metrics. My situation is a bit different than this since I only have two classes, but I'm in essence running the same experiment multiple times, but I could use a similar weighted-averaging approach. Thank you for the pointer. $\endgroup$ – jon_simon Sep 3 '19 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.