26
$\begingroup$

I recently completed a Kaggle competition in which roc auc score was used as per competition requirement. Before this project, I normally used f1 score as the metric to measure model performance. Going forward, I wonder how should I choose between these two metrics? When to use which, and what are their respective pros and cons?

Btw, I read the article here What are the differences between AUC and F1-score?, but it doesn't tell me when to use which.

Thanks in advance for any help!

$\endgroup$
14
$\begingroup$

None of the measures listed here are proper accuracy scoring rules, i.e., rules that are optimized by a correct model. Consider the Brier score and log-likelihood-based measures such as pseudo $R^2$. The $c$-index (AUROC; concordance probability) is not proper but is good for describing a single model. It is not sensitive enough to use for choosing models or comparing even as few as two models.

$\endgroup$
  • $\begingroup$ Thank you for your reply Frank! I need some further clarification please. If we can only choose from ROC AUC and F1 score, which one would you choose and why? What are the pros and cons of both of them? $\endgroup$ – George Liu May 4 '16 at 0:32
  • 2
    $\begingroup$ If you are only allowed to choose from among $c$-index and F1 you are not arguing strongly enough. The gold standard is the log-likelihood, penalized log-likelihood, or Bayesian equivalent (e.g., DIC). Next to that is Brier score. $\endgroup$ – Frank Harrell May 5 '16 at 12:15
  • 2
    $\begingroup$ See citeulike.org/user/harrelfe/article/14321176 ; I've shown this with my own simulations. If the imbalance is not due to oversampling/undersampling you can use any proper scoring rule regardless of imbalance. $\endgroup$ – Frank Harrell Mar 21 '18 at 2:48
  • 1
    $\begingroup$ @FrankHarrell: the link is dead, can you recheck it? $\endgroup$ – SiXUlm Aug 14 at 10:33
13
$\begingroup$

Calculation formula:

  • Precision TP/(TP+FP)
  • Recall: TP/(TP+FN)
  • F1-score: 2/(1/P+1/R)
  • ROC/AUC: TPR=TP/(TP+FN), FPR=FP/(FP+TN)

ROC / AUC is the same criteria and the PR (Precision-Recall) curve (F1-score, Precision, Recall) is also the same criteria.

Real data will tend to have an imbalance between positive and negative samples. This imbalance has large effect on PR but not ROC/AUC.

So in the real world, the PR curve is used more since positive and negative samples are very uneven. The ROC/AUC curve does not reflect the performance of the classifier, but the PR curve can.

If you just do the experiment in research papers, you can use the ROC, the experimental results will be more beautiful. On another hand, PR curve use in the real problem, and it has better interpretability.

$\endgroup$
7
$\begingroup$

Above answers are both good.

But the point I want to point out is AUC (Area under ROC) is problematic especially the data is imbalanced (so called highly skewed: $Skew=\frac{negative\;examples}{positive\;examples}$ is large). This kind of situations is very common in action detection, fraud detection, bankruptcy prediction ect. That is, the positive examples you care have relatively low rates of occurrence.

With imbalanced data, the AUC still gives you specious value around 0.8. However, it is high due to large FP, rather than the large TP (True positive).

Such as the example below,

TP=155,   FN=182
FP=84049, TN=34088

So when you use AUC to measure the performance of classifier, the problem is the increasing of AUC doesn't really reflect a better classifier. It's just the side-effect of too many negative examples. You can simply try in you dataset.

The paper Facing Imbalanced Data Recommendations for the Use of Performance Metrics found "while ROC was unaffected by skew, the precision-recall curves suggest that ROC may mask poor performance in some cases." Searching for a good performance metrics is still a open question. A general F1-score may help $$ F_\beta = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta^2 \cdot \mathrm{precision}) + \mathrm{recall}}$$

where the $\beta$ is the relative importance of precision comparing to recall.

Then, my suggestions for imbalanced data are similar to this post. You can also try the decile table, which can be construct by searching "Two-by-Two Classification and Decile Tables". Meanwhile, I am also studying on this problem and will give better measure.

$\endgroup$
  • $\begingroup$ If you care about the performance of a method, you'd better use ROC to show its classification performance, But if you care more about the actual prediction of true positive, the F1-score is welcome in industry. $\endgroup$ – Xiaorui Zhu Mar 22 '17 at 21:34
  • 2
    $\begingroup$ In a real business setting, costs of false positives and costs of false negative can be estimated. Then the final classification should be based off a probabilistic model and a classification threshold chosen to minimize the cost of false classifications. I don't really think accuracy, or F score have many actual applications for the disciplined data scientist. $\endgroup$ – Matthew Drury May 8 '17 at 4:25
  • $\begingroup$ Yes, I agree with the procedure of decision method that minimize the cost of false classification w.r.t cut-off probability and model. And in some cases, asymmetric cost can be applied to FP and FN. But the point of accuracy and F score is to check the overall performance of a model or compare performance among several models. Indeed, with data in hand as data scientist, cost minimization might be always possible. But I am curious about do data scientist in practical need the distribution(or variation)of the solution of decision problem. I would like to know if you could share some with me.Thx $\endgroup$ – Xiaorui Zhu Jun 13 '17 at 17:52
  • 1
    $\begingroup$ Personally, I would always evaluate the goodness of fit of a model on the basis of the conditional probabilities it predicts. So I would always compare models using a proper scoring rule like log-loss, use bootstrapping to make sure the improvement is not noise, and maybe supplement with AUC. $\endgroup$ – Matthew Drury Jun 13 '17 at 18:13
  • 2
    $\begingroup$ I don't think that is true. AUC is specifically built to be insensitive to class imbalance, I've done extensive simulations on this and found that to be true. Also, when comparing models, they should be build on data sets sampled from the same population, making any issue with class imbalance nill. $\endgroup$ – Matthew Drury Jun 13 '17 at 20:17
4
$\begingroup$

To put in very simple words when you have a data imbalance i.e., the difference between the number of examples you have for positive and negative classes is large, you should always use F1-score. Otherwise you can use ROC/AUC curves.

$\endgroup$
  • $\begingroup$ Your definition of "data imbalance" is such that you'd pretty much always use F1-score, so this isn't much help. Maybe you could expand on this a little? $\endgroup$ – jbowman Aug 4 '18 at 14:20
  • $\begingroup$ I had missed a very important word there...apologies. Edited my response. Let me know if you need more clarification. $\endgroup$ – balboa Aug 4 '18 at 15:07
1
$\begingroup$

For some multi class classification problems, analyzing and visualizing ROC/AUC is not straightforward. You may look into this question, How to plot ROC curves in multiclass classification?. Under such situation, using F1 score could be a better metric.

And F1 score is a common choice for information retrieval problem and popular in industry settings. Here is an well explained example, Building ML models is hard. Deploying them in real business environments is harder.

$\endgroup$
1
$\begingroup$

If the objective of classification is scoring by probability, it is better to use AUC which averages over all possible thresholds. However, if the objective of classification just needs to classify between two possible classes and doesn't require how likely each class is predicted by the model, it is more appropriate to rely on F-score using a particular threshold.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.