3
$\begingroup$

When selecting models in binary classification, there are a couple of most often used metrics, such as accuracy, AUROC, F1 score, logloss and Brier score. I understand what each of those metrics means. But in actual model evaluation, I'm often confused about which is the correct and best one to use. This is critical in model evaluation.

Can someone help clarifying this importance topic? It will be most helpful for me and everyone to truly understand if you can briefly illustrate with some SPECIFIC CASES to correctly use each of those metrics.

Very much appreciate it in advance.

$\endgroup$
  • 2
    $\begingroup$ You have two vary different types of metrics here. Log loss and brier score are used to compare models, while accuracy and f1 score are used to compare decision rules. This is at least part of the answer. $\endgroup$ – Matthew Drury May 23 '18 at 15:53
  • $\begingroup$ I have seen many examples that compare models in hyperparameter tunning using AUROC or accuracy, . Are you suggesting we should always use logloss or brier score? Could you elaborate more on that? @Matthew Drury $\endgroup$ – zesla May 24 '18 at 13:35
  • 1
    $\begingroup$ I'll try to find some time to write a detailed answer, but yah, you should general be using a proper scoring rule like log-loss or brier score to compare your models. AUC makes some sense when you care more about ranking correctness than having well calibrated probabilities, or when your intent is to ultimately develop a decision rule, but are not in control of the costs and benefits of different scenerios. Accuracy is only valid as a last resort, as multi-class problems with many, many classes, where doing something more principled is underdeveloped. $\endgroup$ – Matthew Drury May 24 '18 at 15:28
  • 3
    $\begingroup$ There's lot of historical baggage and inertia in this area, and lots of people just do things without thinking through whether what they are measuring is commensurate with the intent of the model. It's a good question that is worth a detailed answer; forthcoming. $\endgroup$ – Matthew Drury May 24 '18 at 15:29
  • $\begingroup$ Thank you so much for you answer! I definitely agree with you. I just got really confused that so many tutorials use accuracy for binary classification. Once my colleague asked me why not always use logloss. I got stuck and do not know how I should answer that question.... @Matthew Drury $\endgroup$ – zesla May 25 '18 at 1:13
2
$\begingroup$

To elaborate on some of the comments. Most models give a probability output--- say 0.78. If you convert this to a 1, then you have made a "decision". Accuracy, ROC, precision, recall, are all functions of decisions; log likelihood, brier score are functions of probabilities. Often, it's best to leave the decision to the end user, but there are notable exceptions (machine translation, speech recognition (the end user is the computer), self driving cars).

accuracy

Better to not. Perhaps with balanced classes.

AUROC

This just gives sensitivity and specificity as you vary the threshold. Applications include validation in poorly written papers on risk prediction :). Seriously, though, use it if you are interested in providing binary decisions (1 or 0) as output, not a probability estimate. This is important for example when you are guiding a self driving car that must make decisions quickly as to what is road and what is not---and the cost of a false positive is identical to the cost of a false negative.

F1 score

A combination of precision and recall. Application: document retrieval. Generally document retrieval and NLP tasks like machine translation use these metrics because a decision is made (e.g., in machine translation, you want a sentence as output, not some probability distribution over sentences). Precision, in contrast to sensitivity and specificity, depends on prevalence. Therefore it is a good metric for very rare classes, like in document retrieval.

logloss

If you're interested in probability estimates and have a well-specified predictive model. This is the loss function when you minimize the negative log likelihood of a collection of independent Bernoulli random variables. This is related to the probability of your dataset using your estimated parameters. Application: risk score, prediction. Eg, what's the probability of disease or of defaulting on a loan.

Brier score

If you're interested in probability estimates, this is basically a measure of how close (using mean squared error) your estimates are numerically to the actual probabilities. This is nice because it's kind of bounded as opposed to log loss, but I have found it to be kind of insensitive.

$\endgroup$
1
$\begingroup$

Never use accuracy to evaluate a binary classifier. Here’s an extremely accurate prediction you can make at any party you attend: no one present is an astronaut. Because there are about 8 billion people in the world and only about 600 astronauts, this prediction will almost always be 100% accurate. It will also never correctly predict an astronaut. In any binary classification problem, pick the label that occurs most frequently in the training set and your accuracy will be at least 50%.

The metrics to use for binary classification are precision and recall (and in some applications additional metrics like time-to-detect or computational cost). The relative value placed on these metrics depends on the application and the cost of false positives vs false negatives. If you value them equally, F1 is commonly used; otherwise F0.5 or F2 are common too. But really, look at precision and recall separately to understand what’s going on.

Precision and recall work well in theory, and they work well in practice. They’re easy to communicate to stakeholders. Many data-driven CEOs know what precision and recall are and how to reason about them effectively.

Log-loss (cross entropy) is usually best as the score function for training a classifier (binary or multilabel). It measures the relative uncertainty between the classes your model predicts and the true classes. Brier score is sometimes used for non-ordinal multilabel classification, but is in general inferior to log-loss.

If you’re deeply interested in the theory behind this topic, Some Comparisons among Quadratic, Spherical, and Logarithmic Scoring Rules (2007, Bickel) has a gentle introduction. Tl;dr: use log-loss.

$\endgroup$
  • 3
    $\begingroup$ I think your answer would be improved by drawing a clear distinction between evaluating a model, which makes probabilistic predictions, and a decision rule, which assigns observations to classes. Things like accuracy, precision, are useful for evaluating decision rules, but its debatable whether it is wise to use them to evaluate probabilistic predictions. $\endgroup$ – Matthew Drury May 27 '18 at 2:17
  • $\begingroup$ @MatthewDrury I didn’t draw such a distinction because my advice is the same in both cases. Most binary classifieds these days are probabilistic under the hood, but a binary classified is assigning observations to classes, and I would strongly recommend precision and recall for evaluating that assignment and strongly recommend against accuracy for it. $\endgroup$ – Michael Brundage May 27 '18 at 2:28
  • 3
    $\begingroup$ I agree with your point about accuracy, but think it's dangerous to so widely recommend using decision rule based metric to evaluate probabilistic models. There are many, many applications of these models where the evaluation should be based ONLY on the probabilities. For example, if the models are used to quantify risk (as in insurance) precision and recall are irrelevant, if they are used to rank, so as to prioritize interventions to the most likely cases, precision and recall at a fixed threshold are irrelevant. $\endgroup$ – Matthew Drury May 27 '18 at 19:08
  • 3
    $\begingroup$ In other words, there are many applications where the probabilities are not just "under the hood", they are the actual goal of the analysis. I think we've overreached as a community, and are bordering on forgetting this point in how we advise new practitioners. $\endgroup$ – Matthew Drury May 27 '18 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.