Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I've got the results of two classifiers based on 5 different splits of training and testing sets. Their mean and std of the results are as follow:
Method-------Accuracy -- MacroF1 -- BinaryF1---- ROC AUC
Classifier1 0.71±0.032 0.57±0.058 0.33±0.133 0.46±0.095
Classifier2 0.58±0.178 0.44±0.134 0.26±0.137 0.56±0.063
Is this a possible situation? The first three metrics are better in the first classifier and the last one is better for the second classifier. Is this a possible situation? Thanks
Accuracy and F1-type scores depend on the probability prediction that you choose as the cutoff for assignment of a case into a category. It's quite possible that a change of that cutoff (typically a hidden default of p > 0.5 for binary classification, or the highest predicted probability for a multi-category classification) could affect any of those scores. None of those, however, is a good measure of a model's quality.
AUC is much better in that regard. Although it is not a strictly proper scoring rule, it at least covers the entire range of modeled probabilities rather than depending on a particular cutoff. In your case, neither model gives an AUC significantly different from the value of 0.5 that you get just by chance, so you need to develop yet another model in any event.
In fact, none of the other scores differ significantly from each other between the 2 models, when you take the associated errors into account, so it's not even safe to say that "the first three metrics are better in the first classifier."
