I have a dataset with a non-binary target class $c$. I want to compute the AUC of my classifier and can do this easily using the one-vs-rest approach. I train $\binom{n}{2}$ classifiers where n is the number of different values that $c$ can take and compute the AUC for each of those classifiers then just take the mean of different AUCs.

The problem is that sometimes, the AUC is lower than $0.5$. This seems okay to me, since this applies for binary classes and my intuition tells me I should only be worried if the overall AUC was lower than $.25$ when the target class can take 4 different values. Is this logic flawed or is my intuition right?

I've also noticed that the AUC of some of the binary classifiers is lower than $.5$. In this case it should be okay to change it to $1-AUC$ since it's a binary classifier, or will this mess up the general result.

Any insights would be appreciated. Is my approach correct or am I messing up the overall score with my tampering?

  • $\begingroup$ Possible duplicate of How to plot ROC curves in multiclass classification? $\endgroup$
    – EdM
    Dec 12 '16 at 15:46
  • $\begingroup$ No. The question you have provided generally asks how to compute the AUC for multiclass problems. I know how to do this. My question is more theoretical as to what I can do with the actual pairwise AUC scores so I do not mess the overall score up. The question you have provided has very little to do with my question. $\endgroup$
    – Pavlin
    Dec 12 '16 at 16:11
  • $\begingroup$ I'm not sure this is really a duplicate of the linked thread. This asks specifically if it is OK to use 1-AUC in the computation in place of the AUC. I don't see that addressed in the possible duplicate. FWIW, other relevant threads include: Unbalanced dataset - ROC curve to compare classifiers?, & AUC for more than two groups? $\endgroup$ Dec 12 '16 at 16:23
  • 1
    $\begingroup$ You can't get an average <.5 unless some of the component AUCs are <.5, & you shouldn't generally get that in a binary classification problem. Granted, these are coming from the same multiclass classification model, but it might be worth investigating these to see what happened & if a better model is possible. $\endgroup$ Dec 12 '16 at 16:26
  • $\begingroup$ @gung this was my thinking as well. But since it's already the binary classifiers giving me such a terrible AUC, there should be no harm in just flipping the output there, which would give me $1-AUC$ score. I'm just trying to verify this wouldn't somehow mess up my overall score. $\endgroup$
    – Pavlin
    Dec 12 '16 at 17:16

I have found out that I cannot change the AUCs in any way. This would be incorrect since this skews the overall score. If I were to do this, I would have to know in advance (or have a method to determine) which of classifiers would have to be flipped. Doing so after the fact is not okay.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.