# How to choose between ROC AUC and F1 score?

I recently completed a Kaggle competition in which ROC AUC score was used as per competition requirement. Before this project, I normally used F1-score as the metric to measure model performance. Going forward, I wonder how should I choose between these two metrics? When to use which, and what are their respective pros and cons?

I read What are the differences between AUC and F1-score?, but it doesn't tell me when to use which.

None of the measures listed here are proper accuracy scoring rules, i.e., rules that are optimized by a correct model. Consider the Brier score and log-likelihood-based measures such as pseudo $R^2$. The $c$-index (AUROC; concordance probability) is not proper but is good for describing a single model. It is not sensitive enough to use for choosing models or comparing even as few as two models.

• Thank you for your reply Frank! I need some further clarification please. If we can only choose from ROC AUC and F1 score, which one would you choose and why? What are the pros and cons of both of them? Commented May 4, 2016 at 0:32
• If you are only allowed to choose from among $c$-index and F1 you are not arguing strongly enough. The gold standard is the log-likelihood, penalized log-likelihood, or Bayesian equivalent (e.g., DIC). Next to that is Brier score. Commented May 5, 2016 at 12:15
• See citeulike.org/user/harrelfe/article/14321176 ; I've shown this with my own simulations. If the imbalance is not due to oversampling/undersampling you can use any proper scoring rule regardless of imbalance. Commented Mar 21, 2018 at 2:48
• @FrankHarrell: the link is dead, can you recheck it? Commented Aug 14, 2019 at 10:33
• Probably not the paper @FrankHarrell meant, but related is fharrell.com/post/class-damage. Commented Jan 17 at 19:32

Calculation formula：

• Precision: TP/(TP+FP)
• Recall: TP/(TP+FN)
• F1-score： 2/(1/P+1/R)
• ROC/AUC： TPR=TP/(TP+FN), FPR=FP/(FP+TN)

ROC / AUC is the same criteria and the PR (Precision-Recall) curve (F1-score, Precision, Recall) is also the same criteria.

Real data will tend to have an imbalance between positive and negative samples. This imbalance has large effect on PR but not on ROC/AUC.

So in the real world, the PR curve is used more since it is common for positive and negative samples to be very uneven. The ROC/AUC curve does not reflect the performance of the classifier, but the PR curve can.

Use the ROC if you want to report experimental results in research papers, because the experimental results will be more beautiful. On the other hand, use the PR curve in a real-world problem as it has better interpretability.

• I think 'This imbalance has large effect on PR but not ROC/AUC.' may be a bit misleading/unclear.
– 24n8
Commented Aug 30, 2021 at 2:18

But what I want to point out is AUC (Area under ROC) is problematic especially the data is imbalanced (so called highly skewed: $$Skew=\frac{negative\;examples}{positive\;examples}$$ is large). This kind of situations is very common in action detection, fraud detection, bankruptcy prediction ect. That is, the positive examples you care have relatively low rates of occurrence.

With imbalanced data, the AUC still gives you suspicious value around 0.8. However, it is high due to large TN, rather than the large TP (True positive).

Such as the example below,

TP=155,   FN=182
FP=84049, TN=34088


So when you use AUC to measure the performance of classifier, the problem is the increasing of AUC doesn't really reflect a better classifier. It's just the side-effect of too many negative examples. You can simply try in you imbalanced dataset, you will see this issue.

The paper Facing Imbalanced Data Recommendations for the Use of Performance Metrics found "while ROC was unaffected by skew, the precision-recall curves suggest that ROC may mask poor performance in some cases." Searching for a good performance metrics is still a open question. A general F1-score may help $$F_\beta = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta^2 \cdot \mathrm{precision}) + \mathrm{recall}}$$

where the $$\beta$$ is the relative importance of precision comparing to recall.

Then, my suggestions for imbalanced data are similar to this post. You can also try the decile table, which can be construct by searching "Two-by-Two Classification and Decile Tables". Meanwhile, I am also studying on this problem and will give better measure.

## Updates: this might be something you could use in addition to the AUC.

Years later, we developed a new goodness-of-fit measure to resemble the OLS R-squared that can imply the proportion of the variance of the surrogate response explained by explanatory variables. We have had the paper "A new goodness-of-fit measure for probit models: surrogate 𝑅^2" published with all the details. Please check the package webpage as well: https://xiaorui.site/SurrogateRsq/.

• If you care about the performance of a method, you'd better use ROC to show its classification performance, But if you care more about the actual prediction of true positive, the F1-score is welcome in industry. Commented Mar 22, 2017 at 21:34
• In a real business setting, costs of false positives and costs of false negative can be estimated. Then the final classification should be based off a probabilistic model and a classification threshold chosen to minimize the cost of false classifications. I don't really think accuracy, or F score have many actual applications for the disciplined data scientist. Commented May 8, 2017 at 4:25
• Yes, I agree with the procedure of decision method that minimize the cost of false classification w.r.t cut-off probability and model. And in some cases, asymmetric cost can be applied to FP and FN. But the point of accuracy and F score is to check the overall performance of a model or compare performance among several models. Indeed, with data in hand as data scientist, cost minimization might be always possible. But I am curious about do data scientist in practical need the distribution(or variation)of the solution of decision problem. I would like to know if you could share some with me.Thx Commented Jun 13, 2017 at 17:52
• Personally, I would always evaluate the goodness of fit of a model on the basis of the conditional probabilities it predicts. So I would always compare models using a proper scoring rule like log-loss, use bootstrapping to make sure the improvement is not noise, and maybe supplement with AUC. Commented Jun 13, 2017 at 18:13
• I don't think that is true. AUC is specifically built to be insensitive to class imbalance, I've done extensive simulations on this and found that to be true. Also, when comparing models, they should be build on data sets sampled from the same population, making any issue with class imbalance nill. Commented Jun 13, 2017 at 20:17

To put in very simple words when you have a data imbalance i.e., the difference between the number of examples you have for positive and negative classes is large, you should always use F1-score. Otherwise you can use ROC/AUC curves.

• Your definition of "data imbalance" is such that you'd pretty much always use F1-score, so this isn't much help. Maybe you could expand on this a little? Commented Aug 4, 2018 at 14:20
• I had missed a very important word there...apologies. Edited my response. Let me know if you need more clarification. Commented Aug 4, 2018 at 15:07

Despite the less interpretable graph that AUC integrates, the number itself tells you the probability that a randomly chosen positive would be ranked higher than a randomly chosen negative. This is a nice summary of the degree to which positive examples are scored higher than negative examples. If the negatives are ranked higher than all the positives, your AUC is 0. If your negatives are ranked lower than all the positives, the AUC is 1. If the negatives are in the middle or scattered randomly, AUC is around 0.5. Every time your model performance degrades to the point that a positive and negative instance trade ranks when sorted by model score, AUC decreases by a constant number equal to 1/(number of positives x number of negatives).

If you have one negative and 99 positive examples, and that one negative example is ranked higher than all the positive examples, ROC AUC is 0 but you can still achieve a high F1. With a threshold at or lower than your lowest model score (0.5 will work if your model scores everything higher than 0.5), precision and recall are 99% and 100% respectively, leaving your F1 ~99.5%.

In this example, your model performed far worse than a random number generator since it assigned its highest confidence to the only negative example in the dataset. At the same time, it may well be very successful if you care about precision and recall--the problem was so easy even a random number generator could do it!

As a rule of thumb, I've found AUC is useful for comparing models as you're experimenting since it will tell you if you have a bad model despite an easy problem. Precision, recall, F1, and anything that relies on thresholds are useful once you're trying to figure out whether and to what extent it would meet production requirements.

If the objective of classification is scoring by probability, it is better to use AUC which averages over all possible thresholds. However, if the objective of classification just needs to classify between two possible classes and doesn't require how likely each class is predicted by the model, it is more appropriate to rely on F-score using a particular threshold.

Lets start with some formula to see how each measure is calculated (see Wikipedia for a complete list)：

• Precision: $$\frac{TP}{TP+FP}$$
• Recall: $$\frac{TP}{TP+FN}$$
• F1-score$$\frac{2}{\frac{1}{Precision}+\frac{1}{Recall}}=2\times\frac{Precision \times Recall}{Precision + Recall}$$
• AUC curve is built using the following measures:
1. TPR = $$\frac{TP}{TP+FN}$$=Recall
2. FPR = $$\frac{FP}{FP+TN}$$
• PR curve is built using the following measures：
1. Precision
2. Recall

Notice that the AUC is using TPR and FPR criteria. In contrast, the the PR (Precision-Recall) curve and F1-score are using Precision, Recall criteria. Note that the Recall = TPR used in both measures and is identical. So we only focus on the Precision and FPR to see the difference between them.

In general both measures will nicely assess the performance of a classifier. Their difference is pronounced when classes are imbalanced, i.e. when number of samples in the positive class (say class Rare) is very small compared to the negative class (say class Freq). When a classifier is (wrongly) predicting many samples from Freq class as Rare class, then the Precision is going to be small, but FPR could still be large (see the equation above). As a result, the PR curve will more drastically show this lack of performance compared to AUC.

The real-world data tend to be imbalance and often the Rare class is of interest. In such a cases, the using PR curve is recommended. The F1-score produces a single number which is more convenient to work with. So when many classifiers are being compared (or during hyper parameter optimisation), then F1-score is used instead of drawing a PR curve.

The real question is always "how well are the business goals met?". Sometimes that aligns with a standard metric, but not often.

If you are screening drug candidates, and have budget for trials with 10, then what really matters is the probability that there is an excellent candidate among the top ten - which is different to ROC AUC, PR AUC, or F1.

If you are screening patients for cancer, then false negatives are very bad - but you don't want an excessive number of false positives either, because of the related cost and expense. It is always context dependent.

For some multi class classification problems, analyzing and visualizing ROC/AUC is not straightforward. You may look into this question, How to plot ROC curves in multiclass classification?. Under such situation, using F1 score could be a better metric.

And F1 score is a common choice for information retrieval problem and popular in industry settings. Here is an well explained example, Building ML models is hard. Deploying them in real business environments is harder.