Questions tagged [roc]

Receiver Operating Characteristic, also known as the ROC curve.

Filter by
Sorted by
Tagged with
248
votes
6answers
380k views

What does AUC stand for and what is it?

Searched high and low and have not been able to find out what AUC, as in related to prediction, stands for or means.
184
votes
4answers
76k views

ROC vs precision-and-recall curves

I understand the formal differences between them, what I want to know is when it is more relevant to use one vs. the other. Do they always provide complementary insight about the performance of a ...
91
votes
5answers
123k views

How to calculate Area Under the Curve (AUC), or the c-statistic, by hand

I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model. For example, in the validation dataset, I have the true value for the ...
76
votes
4answers
104k views

How to plot ROC curves in multiclass classification?

In other words, instead of having a two class problem I am dealing with 4 classes and still would like to assess performance using AUC.
65
votes
1answer
62k views

Understanding ROC curve

I'm having trouble understanding the ROC curve. Is there any advantage / improvement in area under the ROC curve if I build different models from each unique subset of the training set and use it to ...
55
votes
6answers
61k views

How to determine best cutoff point and its confidence interval using ROC curve in R?

I have the data of a test that could be used to distinguish normal and tumor cells. According to ROC curve it looks good for this purpose (area under curve is 0.9): My questions are: How to ...
34
votes
6answers
37k views

How to choose between ROC AUC and F1 score?

I recently completed a Kaggle competition in which roc auc score was used as per competition requirement. Before this project, I normally used f1 score as the metric to measure model performance. ...
32
votes
4answers
13k views

Optimising for Precision-Recall curves under class imbalance

I have a classification task where I have a number of predictors (one of which is the most informative), and I am using the MARS model to construct my classifier (I am interested in any simple model, ...
31
votes
3answers
30k views

Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

I have two classifiers A: naive Bayesian network B: tree (singly-connected) Bayesian network In terms of accuracy and other measures, A performs comparatively worse than B. However, when I use the R ...
30
votes
4answers
71k views

Area under curve of ROC vs. overall accuracy

I am a little bit confusing about the Area Under Curve (AUC) of ROC and the overall accuracy. Will the AUC be proportional to the overall accuracy? In other words, when we have a larger overall ...
29
votes
3answers
44k views

What is the difference in what AIC and c-statistic (AUC) actually measure for model fit?

Akaike Information Criterion (AIC) and the c-statistic (area under ROC curve) are two measures of model fit for logistic regression. I am having trouble explaining what is going on when the results of ...
27
votes
3answers
28k views

ROC curve for discrete classifiers like SVM: Why do we still call it a “curve”?, Isn't it just a “point”?

In the discussion : how to generate a roc curve for binary classification, I think that the confusion was that a "binary classifier" (which is any classifier that separates 2 classes) was for Yang ...
25
votes
4answers
129k views

Given true positive, false negative rates, can you calculate false positive, true negative?

I have values for True Positive (TP) and False Negative (FN) as follows: ...
23
votes
1answer
3k views

Did I just invent a Bayesian method for analysis of ROC curves?

Preamble This is a long post. If you're re-reading this, please note that I've revised the question portion, though the background material remains the same. Additionally, I believe that I've devised ...
21
votes
4answers
6k views

What is the name of this chart showing false and true positive rates and how is it generated?

The image below shows a continuous curve of false positive rates vs. true positive rates: However, what I don't immediately get is how these rates are being calculated. If a method is applied to a ...
21
votes
2answers
3k views

Adjusting for covariates in ROC curve analysis

This question is about estimating cut-off scores on a multi-dimensional screening questionnaire to predict a binary endpoint, in the presence of correlated scales. I was asked about the interest of ...
20
votes
3answers
9k views

ROC vs Precision-recall curves on imbalanced dataset

I just finished reading this discussion. They argue that PR AUC is better than ROC AUC on imbalanced dataset. For example, we have 10 samples in test dataset. 9 samples are positive and 1 is ...
18
votes
3answers
20k views

Can AUC-ROC be between 0-0.5?

Can AUC-ROC values be between 0-0.5? Does the model ever output values between 0 and 0.5?
18
votes
3answers
16k views

Difference between regression analysis and curve fitting

Can anybody please explain to me the real difference(s) between regression analysis and curve fitting (linear and nonlinear), with an example if possible? It seems that both try to find a ...
17
votes
3answers
14k views

Area under the ROC curve or area under the PR curve for imbalanced data?

I have some doubts about which performance measure to use, area under the ROC curve (TPR as a function of FPR) or area under the precision-recall curve (precision as a function of recall). My data is ...
17
votes
2answers
8k views

Accuracy vs. area under the ROC curve

I constructed an ROC curve for a diagnostic system. The area under the curve was then non-parametrically estimated to be AUC = 0.89. When I tried to calculate the accuracy at the optimum threshold ...
17
votes
2answers
19k views

Average ROC for repeated 10-fold cross validation with probability estimates

I am planning to use repeated (10 times) stratified 10-fold cross validation on about 10,000 cases using machine learning algorithm. Each time the repetition will be done with different random ...
17
votes
1answer
785 views

What does it mean that AUC is a semi-proper scoring rule?

A proper scoring rule is a rule that is maximized by a 'true' model and it doesn't allow 'hedging' or gaming the system (deliberately reporting different results as is the true belief of the model to ...
15
votes
2answers
35k views

Is the Dice coefficient the same as accuracy?

I come across the Dice coefficient for volume similarity (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) and accuracy (https://en.wikipedia.org/wiki/Accuracy_and_precision). ...
15
votes
4answers
5k views

Advantages of ROC curves

What is the advantages of the ROC curves? For example I am classifying some images which is a binary classification problem. I extracted about 500 features and applied a features selection algorithm ...
15
votes
3answers
8k views

How to derive the probabilistic interpretation of the AUC?

Why is the area under the ROC curve the probability that a classifier will rank a randomly chosen "positive" instance (from the retrieved predictions) higher than a randomly chosen "positive" one (...
15
votes
3answers
4k views

Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity

I have a test set of 100 cases and two classifiers. I generated predictions and computed ROC AUC, sensitivity and specificity for both classifiers. Question 1: How can I compute p-value to check if ...
15
votes
2answers
1k views

Combining classifiers by flipping a coin

I am studying a machine learning course and the lecture slides contain information what I find contradicting with the recommended book. The problem is the following: there are three classifiers: ...
15
votes
2answers
6k views

How to do cross-validation with a Cox proportional hazards model?

Suppose I have constructed a prediction model for the occurrence of a particular disease in one dataset (the model building dataset) and now want to check how well the model works in a new dataset (...
14
votes
4answers
1k views

Dichotomizing continuous variables at their optimal cut-off for clinical interpretation

In medical context, when presenting results from a binary outcome with a continuous predictor, the OR (odds ratio) can be difficult to interpret. Example: A doctor does a study in which he wants to ...
14
votes
5answers
2k views

Philosophical question on logistic regression: why isn't the optimal threshold value trained?

Usually in logistic regression, we fit a model and get some predictions on the training set. We then cross-validate on those training predictions (something like here) and decide the optimal threshold ...
14
votes
4answers
54k views

How to interpret a ROC curve?

I applied logistic regression to my data on SAS and here are the ROC curve and classification table. I am comfortable with the figures in the classification table, but not exactly sure what the roc ...
14
votes
2answers
38k views

Using the caret package is it possible to obtain confusion matrices for specific threshold values?

I've obtained a logistic regression model (via train) for a binary response, and I've obtained the logistic confusion matrix via ...
14
votes
4answers
19k views

ROC and multiROC analysis: how to calculate optimal cutpoint?

I'm trying to understand how to compute the optimal cut-point for a ROC curve (the value at which the sensitivity and specificity are maximized). I'm using the dataset ...
14
votes
1answer
22k views

What is the difference between GINI and AUC curve interpretation?

we used to create GINI curve using lift created with help of percentage of good and bad for scorecard modelling. But what I have studied that ROC curve is created using Confusion matrix with ...
14
votes
1answer
3k views

Why is ROC AUC equivalent to the probability that two randomly-selected samples are correctly ranked? [duplicate]

I found there are two ways to understand what AUC stands for but I couldn't get why these two interpretations are equivalent mathematically. In the first interpretation, AUC is the area under the ...
13
votes
2answers
35k views

How we can draw an ROC curve for decision trees?

Normally we cannot draw an ROC curve for the discrete classifiers like decision trees. Am I right? Is there any way to draw an ROC curve for Dtrees?
13
votes
4answers
16k views

In R how to compute the p-value for area under ROC

I struggle to find a way to compute the p-value for the area under a receiver operator characteristic (ROC). I have a continuous variable and a diagnostic test result. I want to see if AUROC is ...
13
votes
1answer
4k views

Comparisson of two models when the ROC curves cross each other

One common measure used to compare two or more classification models is to use the area under the ROC curve (AUC) as a way to indirectly assess their performance. In this case a model with a larger ...
13
votes
1answer
2k views

Connections between $d^\prime$ (d-prime) and AUC (Area Under the ROC Curve); underlying assumptions

In machine learning we may use the area under the ROC curve (often abbreviated AUC, or AUROC) to summarise how well a system can discriminate between two categories. In signal detection theory often ...
13
votes
1answer
23k views

Evaluating a logistic regression model

I've been working on a logistic model and I'm having some difficulties evaluating the results. My model is a binomial logit. My explanatory variables are: a categorical variable with 15 levels, a ...
12
votes
5answers
5k views

What do ROC curves tell you that traditional inference wouldn't?

When would you tend to use ROC curves over some other tests to determine the predictive ability of some measurement on an outcome? When dealing with discrete outcomes (alive/dead, present/absent), ...
12
votes
3answers
3k views

ROC curve crossing the diagonal

I am running a binary classifier at the moment. When I plot the ROC curve I get a good lift at the beginning then it changes direction and crosses the diagonal then of course back up, making the curve ...
11
votes
4answers
5k views

Is AUC the probability of correctly classifying a randomly selected instance from each class?

I read this caption in a paper and have never seen AUC described in this way anywhere else. Is this true? Is there a proof or simple way to see this? Fig. 2 shows the prediction accuracy of ...
11
votes
2answers
13k views

d prime with 100% hit rate probability and 0% false alarm probability

I would like to calculate d prime for a memory task that involves detecting old and new items. The problem I have is that some of the subjects have hit rate of 1 and/or false alarm rate of 0, which ...
11
votes
1answer
9k views

Interpretation of the area under the PR curve

I'm currently comparing three methods and I have the Accuracy, auROC and auPR as metrics. And I have the following results : Method A - acc: 0.75, auROC: 0.75, auPR: 0.45 Method B - acc: 0.65, auROC:...
11
votes
1answer
3k views

Evaluation of classifiers: learning curves vs ROC curves

I would like to compare 2 different classifiers for a multiclass text classification problem that use large training datasets. I am doubting whether I should use ROC curves or learning curves to ...
11
votes
3answers
8k views

“Good” classifier destroyed my Precision-Recall curve. What happened?

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and ...
11
votes
1answer
16k views

How to choose the cutoff probability for a rare event Logistic Regression

I have 100,000 observations (9 dummy indicator variables) with 1000 positives. Logistic Regression should work fine in this case but the cutoff probability puzzles me. In common literature, we ...
10
votes
1answer
3k views

ROC curve drawbacks

In the class yesterday, we were taught about logistic and subsequently the ROC curve and how to use it. My questions are: Is this the most common way to identify if the logistic model is the best? ...

1
2 3 4 5
14