I've got the results of two classifiers based on 5 different splits of training and testing sets. Their mean and std of the results are as follow:

Method-------Accuracy -- MacroF1 -- BinaryF1---- ROC AUC

Classifier1 0.71±0.032 0.57±0.058 0.33±0.133 0.46±0.095

Classifier2 0.58±0.178 0.44±0.134 0.26±0.137 0.56±0.063

Is this a possible situation? The first three metrics are better in the first classifier and the last one is better for the second classifier. Is this a possible situation? Thanks

  • $\begingroup$ @Calimo Great Thanks. $\endgroup$
    – user137927
    Oct 17, 2020 at 18:35

1 Answer 1


Accuracy and F1-type scores depend on the probability prediction that you choose as the cutoff for assignment of a case into a category. It's quite possible that a change of that cutoff (typically a hidden default of p > 0.5 for binary classification, or the highest predicted probability for a multi-category classification) could affect any of those scores. None of those, however, is a good measure of a model's quality.

AUC is much better in that regard. Although it is not a strictly proper scoring rule, it at least covers the entire range of modeled probabilities rather than depending on a particular cutoff. In your case, neither model gives an AUC significantly different from the value of 0.5 that you get just by chance, so you need to develop yet another model in any event.

In fact, none of the other scores differ significantly from each other between the 2 models, when you take the associated errors into account, so it's not even safe to say that "the first three metrics are better in the first classifier."

  • 1
    $\begingroup$ Thanks for your answer. I agree that they're not good models, but what I want to know is that is it possible to get these results, or am I getting something wrong? $\endgroup$
    – user137927
    Oct 16, 2020 at 19:50
  • 2
    $\begingroup$ His point, which I do think he should make explicit in the answer, is that you could try different thresholds. Depending on the threshold, your inferior model may become the superior model in terms of accuracy (which, surprisingly, is a problematic metric). $\endgroup$
    – Dave
    Oct 16, 2020 at 19:54
  • $\begingroup$ @Dave thanks for the suggestion. $\endgroup$
    – EdM
    Oct 16, 2020 at 20:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.