Skip to main content
47 votes

Can AUC-ROC be between 0-0.5?

I am sorry, but these answers are dangerously wrong. No, you cannot just flip AUC after you see the data. Imagine you are buying stocks, and you always bought the wrong one, but you said to yourself, ...
rep_ho's user avatar
  • 7,629
38 votes
Accepted

Can AUC-ROC be between 0-0.5?

A perfect predictor gives an AUC-ROC score of 1, a predictor which makes random guesses has an AUC-ROC score of 0.5. If you get a score of 0 that means the classifier is perfectly incorrect, it is ...
Hugh's user avatar
  • 4,049
35 votes

ROC vs precision-and-recall curves

There is a lot of misunderstanding about evaluation. Part of this comes from the Machine Learning approach of trying to optimize algorithms on datasets, with no real interest in the data. In a ...
David M W Powers's user avatar
35 votes

What does it mean that AUC is a semi-proper scoring rule?

Let's start with an example. Say Alice is a track coach and wants to pick an athlete to represent the team in an upcoming sporting event, a 200m sprint. Naturally she wants to pick the fastest runner. ...
usεr11852's user avatar
  • 44.4k
35 votes
Accepted

What is the formula to calculate the area under the ROC curve from a contingency table?

In the general case: you can't The ROC curve shows how sensitivity and specificity varies at every possible threshold. A contingency table has been calculated at a single threshold and information ...
Calimo's user avatar
  • 3,674
34 votes

Is the Dice coefficient the same as accuracy?

The Dice coefficient (also known as Dice similarity index) is the same as the F1 score, but it's not the same as accuracy. The main difference might be the fact that accuracy takes into account true ...
Dvir Berebi's user avatar
34 votes

ROC vs Precision-recall curves on imbalanced dataset

First, the claim on the Kaggle post is bogus. The paper they reference, "The Relationship Between Precision-Recall and ROC Curves", never claims that PR AUC is better than ROC AUC. They simply compare ...
Calimo's user avatar
  • 3,674
24 votes

How to choose between ROC AUC and F1 score?

None of the measures listed here are proper accuracy scoring rules, i.e., rules that are optimized by a correct model. Consider the Brier score and log-likelihood-based measures such as pseudo $R^2$. ...
Frank Harrell's user avatar
23 votes

How to choose between ROC AUC and F1 score?

Calculation formula: Precision: TP/(TP+FP) Recall: TP/(TP+FN) F1-score: 2/(1/P+1/R) ROC/AUC: TPR=TP/(TP+FN), FPR=FP/(FP+TN) ROC / AUC is the same criteria and the PR (Precision-Recall) curve (F1-...
WeiYuan's user avatar
  • 516
22 votes
Accepted

Comparison of two models when the ROC curves cross each other

A ROC curve visualizes TPR and FPR for all possible thresholds. If you plot two ROC curves 'A' and 'B' and they do not cross each other, then one of your classifiers is clearly performing better, ...
Laksan Nathan's user avatar
22 votes

Where in the ROC curve does it tell you what the threshold is?

Each (FPR, TPR) point on a ROC curve is associated with a threshold. However, the thresholds are not typically drawn on the curve itself. It is possible to reveal them, either adding extra annotation ...
Calimo's user avatar
  • 3,674
21 votes

What is the difference between GINI and AUC curve interpretation?

The Gini Coefficient is the summary statistic of the Cumulative Accuracy Profile (CAP) chart. It is calculated as the quotient of the area which the CAP curve and diagonal enclose and the ...
the_owl's user avatar
  • 361
21 votes

How to derive the probabilistic interpretation of the AUC?

From Measuring classifier performance: a coherent alternative to the area under the ROC curve : First thing, let's try to define the area under the ROC curve formally. Some assumptions and ...
alebu's user avatar
  • 419
21 votes
Accepted

Philosophical question on logistic regression: why isn't the optimal threshold value trained?

A threshold isn't trained with the model because logistic regression isn't a classifier (cf., Why isn't Logistic Regression called Logistic Classification?). It is a model to estimate the parameter, $...
gung - Reinstate Monica's user avatar
20 votes

What does AUC stand for and what is it?

The answers in this forum are great and I come back here often for reference. However, one thing was always missing. From @Frank's answer, we see the interpretation of AUC as the probability that a ...
ryu576's user avatar
  • 2,540
19 votes

How to derive the probabilistic interpretation of the AUC?

@alebu's answer is great. But its notation is nonstandard and uses 0 for the positive class and 1 for the negative class. Below are the results for the standard notation (0 for the negative class and ...
Lei Huang's user avatar
  • 956
19 votes
Accepted

AUPRC vs. AUC-ROC?

ROC AUC is the area under the curve where x is false positive rate (FPR) and y is true positive rate (TPR). PR AUC is the area under the curve where x is recall and y is precision. recall = TPR = ...
Sam Weisenthal's user avatar
19 votes

Determine how good an AUC is (Area under the Curve of ROC)

From the comments: Calimo: If you are a trader and you can get an AUC of 0.501 in predicting future financial transactions, you're the richest man in the world. If you are a CPU engineer and your ...
18 votes

How to determine the optimal threshold for a classifier and generate ROC curve?

The choice of a threshold depends on the importance of TPR and FPR classification problem. For example, if your classifier will decide which criminal suspects will receive a death sentence, false ...
Bananin's user avatar
  • 728
18 votes
Accepted

Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity

Wojtek J. Krzanowski and David J. Hand ROC Curves for Continuous Data (2009) is a great reference for all things related to ROC curves. It collects together a number of results in what is a ...
Sycorax's user avatar
  • 91.4k
17 votes
Accepted

Is AUC the probability of correctly classifying a randomly selected instance from each class?

The quotation is slightly incorrect. The correct statement is that ROC AUC is the probability a randomly-chosen positive example is ranked more highly than a randomly-chosen negative example. This is ...
Sycorax's user avatar
  • 91.4k
17 votes
Accepted

AUC and class imbalance in training/test dataset

It depends how you mean the word sensitive. The ROC AUC is sensitive to class imbalance in the sense that when there is a minority class, you typically define this as the positive class and it will ...
David Ernst's user avatar
  • 3,171
17 votes

ROC vs Precision-recall curves on imbalanced dataset

Your example is definitely correct. However, I think in the context of Kaggle competition / real life application, a skewed dataset usually means a dataset with much less positive samples than ...
user2512796's user avatar
17 votes

Can AUC-ROC be between 0-0.5?

They can, if the system you're analyzing performs below chance level. Trivially, you could easily construct a classifier with 0 AUC by having it always answer opposite to the truth. In practice of ...
Ruben van Bergen's user avatar
17 votes
Accepted

What is the origin of the "receiver operating characteristic" (ROC) terminology?

The earliest book reference that I know of is Woodward, P. M. (1953). Probability and information theory with applications to radar. London: Pergamon Press. but the concept, which was developed ...
Dilip Sarwate's user avatar
17 votes
Accepted

Dichotomizing continuous variables at their optimal cut-off for clinical interpretation

Dichotomizing a continuous covariate is ill-advised, as has been noted by other users. One strategy I employ is to rescale the predictor to something more reasonable. 1 mmHg may not be a very ...
Demetri Pananos's user avatar
16 votes

How to determine the optimal threshold for a classifier and generate ROC curve?

Choose the point closest to the top left corner of your ROC space. Now the threshold used to generate this point should be the optimal one.
dr_rk's user avatar
  • 269
16 votes

Area under the ROC curve when there is imbalance: is there a problem, and if not, why does this rumor exist?

This is actually a very simple issue. The area under the ROC curve (AUROC) equals the Wilcoxon-Mann-Whitney-Somers concordance probability, a $U$-statistic, i.e., take all possible pairs of an ...
Frank Harrell's user avatar
15 votes

Is up- or down-sampling imbalanced data actually that effective? Why?

The short answer appears to be Yes: there is some evidence that upsampling of the minority class and/or downsampling of the majority class in a training set can somewhat improve out-of-sample AUC (...
Jake Westfall's user avatar
15 votes

ROC vs precision-and-recall curves

TL;DR $AUC_{PvR}$ highlights the amount of False Positives relative to the class size, whereas $AUC_{ROC}$ better reflects the total amount of False Positives independent of in which class they come ...
0-_-0's user avatar
  • 289

Only top scored, non community-wiki answers of a minimum length are eligible