How to choose between ROC AUC and F1 score?

I recently completed a Kaggle competition in which roc auc score was used as per competition requirement. Before this project, I normally used f1 score as the metric to measure model performance. Going forward, I wonder how should I choose between these two metrics? When to use which, and what are their respective pros and cons?

Btw, I read the article here What are the differences between AUC and F1-score?, but it doesn't tell me when to use which.

Thanks in advance for any help!

Calculation formula：

• Precision TP/(TP+FP)
• Recall: TP/(TP+FN)
• F1-score： 2/(1/P+1/R)
• ROC/AUC： TPR=TP/(TP+FN), FPR=FP/(FP+TN)

ROC / AUC is the same criteria and the PR (Precision-Recall) curve (F1-score, Precision, Recall) is also the same criteria.

Real data will tend to have an imbalance between positive and negative samples. This imbalance has large effect on PR but not ROC/AUC.

So in the real world, the PR curve is used more since positive and negative samples are very uneven. The ROC/AUC curve does not reflect the performance of the classifier, but the PR curve can.

If you just do the experiment in research papers, you can use the ROC, the experimental results will be more beautiful. On another hand, PR curve use in the real problem, and it has better interpretability.

• I think 'This imbalance has large effect on PR but not ROC/AUC.' may be a bit misleading/unclear.
– 24n8
Aug 30, 2021 at 2:18

None of the measures listed here are proper accuracy scoring rules, i.e., rules that are optimized by a correct model. Consider the Brier score and log-likelihood-based measures such as pseudo $R^2$. The $c$-index (AUROC; concordance probability) is not proper but is good for describing a single model. It is not sensitive enough to use for choosing models or comparing even as few as two models.

• Thank you for your reply Frank! I need some further clarification please. If we can only choose from ROC AUC and F1 score, which one would you choose and why? What are the pros and cons of both of them? May 4, 2016 at 0:32
• If you are only allowed to choose from among $c$-index and F1 you are not arguing strongly enough. The gold standard is the log-likelihood, penalized log-likelihood, or Bayesian equivalent (e.g., DIC). Next to that is Brier score. May 5, 2016 at 12:15
• See citeulike.org/user/harrelfe/article/14321176 ; I've shown this with my own simulations. If the imbalance is not due to oversampling/undersampling you can use any proper scoring rule regardless of imbalance. Mar 21, 2018 at 2:48
• @FrankHarrell: the link is dead, can you recheck it? Aug 14, 2019 at 10:33

But what I want to point out is AUC (Area under ROC) is problematic especially the data is imbalanced (so called highly skewed: $$Skew=\frac{negative\;examples}{positive\;examples}$$ is large). This kind of situations is very common in action detection, fraud detection, bankruptcy prediction ect. That is, the positive examples you care have relatively low rates of occurrence.

With imbalanced data, the AUC still gives you specious value around 0.8. However, it is high due to large FP, rather than the large TP (True positive).

Such as the example below,

TP=155,   FN=182
FP=84049, TN=34088


So when you use AUC to measure the performance of classifier, the problem is the increasing of AUC doesn't really reflect a better classifier. It's just the side-effect of too many negative examples. You can simply try in you imbalanced dataset, you will see this issue.

The paper Facing Imbalanced Data Recommendations for the Use of Performance Metrics found "while ROC was unaffected by skew, the precision-recall curves suggest that ROC may mask poor performance in some cases." Searching for a good performance metrics is still a open question. A general F1-score may help $$F_\beta = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta^2 \cdot \mathrm{precision}) + \mathrm{recall}}$$

where the $$\beta$$ is the relative importance of precision comparing to recall.

Then, my suggestions for imbalanced data are similar to this post. You can also try the decile table, which can be construct by searching "Two-by-Two Classification and Decile Tables". Meanwhile, I am also studying on this problem and will give better measure.

• If you care about the performance of a method, you'd better use ROC to show its classification performance, But if you care more about the actual prediction of true positive, the F1-score is welcome in industry. Mar 22, 2017 at 21:34
• In a real business setting, costs of false positives and costs of false negative can be estimated. Then the final classification should be based off a probabilistic model and a classification threshold chosen to minimize the cost of false classifications. I don't really think accuracy, or F score have many actual applications for the disciplined data scientist. May 8, 2017 at 4:25
• Yes, I agree with the procedure of decision method that minimize the cost of false classification w.r.t cut-off probability and model. And in some cases, asymmetric cost can be applied to FP and FN. But the point of accuracy and F score is to check the overall performance of a model or compare performance among several models. Indeed, with data in hand as data scientist, cost minimization might be always possible. But I am curious about do data scientist in practical need the distribution(or variation)of the solution of decision problem. I would like to know if you could share some with me.Thx Jun 13, 2017 at 17:52
• Personally, I would always evaluate the goodness of fit of a model on the basis of the conditional probabilities it predicts. So I would always compare models using a proper scoring rule like log-loss, use bootstrapping to make sure the improvement is not noise, and maybe supplement with AUC. Jun 13, 2017 at 18:13
• I don't think that is true. AUC is specifically built to be insensitive to class imbalance, I've done extensive simulations on this and found that to be true. Also, when comparing models, they should be build on data sets sampled from the same population, making any issue with class imbalance nill. Jun 13, 2017 at 20:17

To put in very simple words when you have a data imbalance i.e., the difference between the number of examples you have for positive and negative classes is large, you should always use F1-score. Otherwise you can use ROC/AUC curves.

• Your definition of "data imbalance" is such that you'd pretty much always use F1-score, so this isn't much help. Maybe you could expand on this a little? Aug 4, 2018 at 14:20
• I had missed a very important word there...apologies. Edited my response. Let me know if you need more clarification. Aug 4, 2018 at 15:07

If the objective of classification is scoring by probability, it is better to use AUC which averages over all possible thresholds. However, if the objective of classification just needs to classify between two possible classes and doesn't require how likely each class is predicted by the model, it is more appropriate to rely on F-score using a particular threshold.

Despite the less interpretable graph that AUC integrates, the number itself tells you the probability that a randomly chosen positive would be ranked higher than a randomly chosen negative. This is a nice summary of the degree to which positive examples are scored higher than negative examples. If the negatives are ranked higher than all the positives, your AUC is 0. If your negatives are ranked lower than all the positives, the AUC is 1. If the negatives are in the middle or scattered randomly, AUC is around 0.5. Every time your model performance degrades to the point that a positive and negative instance trade ranks when sorted by model score, AUC decreases by a constant number equal to 1/(number of positives x number of negatives).

If you have one negative and 99 positive examples, and that one negative example is ranked higher than all the positive examples, ROC AUC is 0 but you can still achieve a high F1. With a threshold at or lower than your lowest model score (0.5 will work if your model scores everything higher than 0.5), precision and recall are 99% and 100% respectively, leaving your F1 ~99.5%.

In this example, your model performed far worse than a random number generator since it assigned its highest confidence to the only negative example in the dataset. At the same time, it may well be very successful if you care about precision and recall--the problem was so easy even a random number generator could do it!

As a rule of thumb, I've found AUC is useful for comparing models as you're experimenting since it will tell you if you have a bad model despite an easy problem. Precision, recall, F1, and anything that relies on thresholds are useful once you're trying to figure out whether and to what extent it would meet production requirements.

For some multi class classification problems, analyzing and visualizing ROC/AUC is not straightforward. You may look into this question, How to plot ROC curves in multiclass classification?. Under such situation, using F1 score could be a better metric.

And F1 score is a common choice for information retrieval problem and popular in industry settings. Here is an well explained example, Building ML models is hard. Deploying them in real business environments is harder.