I read literature showing how metrics such Accuracy, ROC-AUC, F-1 score can be misleading for the evaluation of classifiers on imbalanced data. Is G-mean score suitable for this context?

  • 1
    $\begingroup$ Where did you read that about ROC-AUC? Maybe it is not great in comparison with the Brier score but it is not outright misleading. Accuracy on the other hand, yes, it can be misleading. G-mean (based either on Precision/Recall or Specif/Sensitivity) will be "as good as" F1-score at best. In the absence of any more context sticking with ROC-AUC and ROC-PR as you seem to do already is the safe option (or use Brier score). $\endgroup$ – usεr11852 Apr 10 at 3:40
  • $\begingroup$ @usεr11852 the references in this blog dpmartin42.github.io/posts/r/imbalanced-classes-part-2 $\endgroup$ – user000 Apr 10 at 20:44
  • $\begingroup$ That blog raises the valid point that ROC-PR can be more informative than PR-AUC (as I also suggest looking at in my comment). It says nothing about the G-mean. $\endgroup$ – usεr11852 Apr 10 at 21:32
  • $\begingroup$ @usεr11852 That blog and provided references clearly illustrate the opposite: PR-AUC is actually more informative for imbalanced classification than ROC-AUC; ROC-AUC can be misleading for those cases as shown in provided examples. Thanks for the insight on G-mean though. I read about Brier score being ill-suited to compare different classifiers in imbalanced cases, only being useful for comparing modifications (as in changed hyper-parameters) of the same classifier, but I lost the reference somewhere. If I do find it, I will link it here. $\endgroup$ – user000 Apr 11 at 10:06
  • $\begingroup$ Sorry extensive typos in my previous comment. Should read "PR-AUC can be more informative than ROC-AUC". I did not read it through. (I managed to mangle the order as well as make a nonexistent metric in one sentence) $\endgroup$ – usεr11852 Apr 11 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.