4
$\begingroup$

I'm dealing with an image classification problem, with a multiclass imbalanced dataset (the bigger class has 4000 samples and the smaller has 110 samples) with 50 classes and 24000 samples.

I'm using a neural network for doing the task. I'm adopting a set o metrics for evaluating the model. Below I provide the metrics with the values achieved by my model:

  • accuracy: 0.93
  • macro-f1: 0.86
  • weighted f1: 0.93
  • macro precision: 0.87
  • weight precision: 0.93
  • macro recall: 0.85
  • weighted recall: 0.93
  • macro AUROC: 0.99
  • weighted AUROC: 0.99

These values are averages obtained in a 5-fold cross-validation process. That is, the metrics are obtained by classifying each test fold. After that, the values of each metric are averaged.

However, I'm not confident about the values for AUROC (macro and weighted). I think that these values are "too good to be correct".

What do you think? Can you provide some guidelines for ensuring that this is correct for finding my error? Is AUROC suiable for evaluating the model in this context?

$\endgroup$
3
  • $\begingroup$ Can you verify the correctness of some of the easier metrics to check? e.g that the macro recall is 0.85? To be clear, AUCROC can be rather high in imbalanced cases; it is still a valid metric just we actual number might be conceptually high for us. $\endgroup$
    – usεr11852
    Apr 8, 2022 at 9:22
  • $\begingroup$ I think there's potentially a lot of confusion here, and your problem is still not outlined well enough for anyone to give an actual answer. First of all, an ROC curve really just tells you where to place your confidence threshold. A high AUROC just means that you you can better win the game of balancing precision and recall with your confidence threshold. That being said, I do agree that your AUROC seems way too high for the precision and recall numbers you are putting up. How is it calculated, where is the plot? This may be a coding issue rather than a stats issue. $\endgroup$ Apr 8, 2022 at 15:51
  • $\begingroup$ It might help to know what the per-class statistics (without averaging) are. $\endgroup$ Apr 8, 2022 at 17:41

2 Answers 2

4
$\begingroup$

THIS APPLIED BEFORE THE QUESTION EDIT. I will leave it here as a reference for the situation where performance is worse than chance.

Your accuracy is $93\%$. If you just classify everything as the majority class, you score $97\%$, so your error rate is more than double this naïve guessing of everything being in the majority class. Something in your model is amiss. I suspect at least part of it has to do with your use of a complex neural network on a sample size of only $4110$. That sounds like a recipe for overfitting and creating a model whose out-of-sample performance is worse than chance.

My guess is that your software does not give $AUC<0.5$ and is reversing the probabilities, meaning that your true $AUC\approx 0.01$, if you base your $AUC$ calculation on the true outputs of the neural network.

To test this out, you could write some ROC curve plotting code from scratch, using the exact probability values outputted by the neural network. If you do it yourself, then you know for sure that the true $AUC$ is awful.

A crude implementation would be to loop over $0$, $0.01$, $0.02$,$\dots$, $0.98$ , $0.99$, $1$ and calculate the sensitivity and specificity with each such value as the threshold. Then plot the sensitivity values on the vertical axis and one minus the specificity values on the horizontal axis.

$\endgroup$
2
  • $\begingroup$ I'm sorry. I forgot to say that I have 50 classes and 24000 samples in the total. $\endgroup$
    – Zaratruta
    Apr 8, 2022 at 5:05
  • 1
    $\begingroup$ This needs some updating, 4k/24K =16.7% Accuracy. $\endgroup$
    – usεr11852
    Apr 8, 2022 at 9:12
1
$\begingroup$

It may be that your minority class(es) have poor performance in the hard classification (obtained by choosing the class with the largest predicted probability), but that their predicted probabilities rank-order well. The former would explain low macro-averaged hard classification scores compared to high weighted-averaged scores, and the latter would explain why AUROC is not so encumbered.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.