AUROC too high in image classification

Question

I'm dealing with an image classification problem, with a multiclass imbalanced dataset (the bigger class has 4000 samples and the smaller has 110 samples) with 50 classes and 24000 samples.

I'm using a neural network for doing the task. I'm adopting a set o metrics for evaluating the model. Below I provide the metrics with the values achieved by my model:

• accuracy: 0.93
• macro-f1: 0.86
• weighted f1: 0.93
• macro precision: 0.87
• weight precision: 0.93
• macro recall: 0.85
• weighted recall: 0.93
• macro AUROC: 0.99
• weighted AUROC: 0.99

These values are averages obtained in a 5-fold cross-validation process. That is, the metrics are obtained by classifying each test fold. After that, the values of each metric are averaged.

However, I'm not confident about the values for AUROC (macro and weighted). I think that these values are "too good to be correct".

What do you think? Can you provide some guidelines for ensuring that this is correct for finding my error? Is AUROC suiable for evaluating the model in this context?

Answer 1

THIS APPLIED BEFORE THE QUESTION EDIT. I will leave it here as a reference for the situation where performance is worse than chance.

Your accuracy is $93\%$. If you just classify everything as the majority class, you score $97\%$, so your error rate is more than double this naïve guessing of everything being in the majority class. Something in your model is amiss. I suspect at least part of it has to do with your use of a complex neural network on a sample size of only $4110$. That sounds like a recipe for overfitting and creating a model whose out-of-sample performance is worse than chance.

My guess is that your software does not give $AUC<0.5$ and is reversing the probabilities, meaning that your true $AUC\approx 0.01$, if you base your $AUC$ calculation on the true outputs of the neural network.

To test this out, you could write some ROC curve plotting code from scratch, using the exact probability values outputted by the neural network. If you do it yourself, then you know for sure that the true $AUC$ is awful.

A crude implementation would be to loop over $0$, $0.01$, $0.02$,$\dots$, $0.98$ , $0.99$, $1$ and calculate the sensitivity and specificity with each such value as the threshold. Then plot the sensitivity values on the vertical axis and one minus the specificity values on the horizontal axis.

Answer 2

It may be that your minority class(es) have poor performance in the hard classification (obtained by choosing the class with the largest predicted probability), but that their predicted probabilities rank-order well. The former would explain low macro-averaged hard classification scores compared to high weighted-averaged scores, and the latter would explain why AUROC is not so encumbered.