Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.

Filter by
Sorted by
Tagged with
13
votes
3answers
3k views

Is my model any good, based on the diagnostic metric ($R^2$/ AUC/ accuracy/ RMSE etc.) value?

I've fitted my model and am trying to understand whether it's any good. I've calculated the recommended metrics to assess it ($R^2$/ AUC / accuracy / prediction error / etc) but do not know how to ...
90
votes
5answers
122k views

How to calculate Area Under the Curve (AUC), or the c-statistic, by hand

I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model. For example, in the validation dataset, I have the true value for the ...
243
votes
6answers
376k views

What does AUC stand for and what is it?

Searched high and low and have not been able to find out what AUC, as in related to prediction, stands for or means.
30
votes
3answers
29k views

Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

I have two classifiers A: naive Bayesian network B: tree (singly-connected) Bayesian network In terms of accuracy and other measures, A performs comparatively worse than B. However, when I use the R ...
14
votes
3answers
3k views

Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity

I have a test set of 100 cases and two classifiers. I generated predictions and computed ROC AUC, sensitivity and specificity for both classifiers. Question 1: How can I compute p-value to check if ...
11
votes
2answers
2k views

Compare classifiers based on AUROC or accuracy?

I have a binary classification problem and I experiment different classifiers on it: I want to compare the classifiers. which one is a better measure AUC or accuracy? And why? ...
20
votes
3answers
4k views

Why AUC =1 even classifier has misclassified half of the samples?

I am using a classifier which returns probabilities. To calculate AUC, I am using pROC R-package. The output probabilities from classifier are: ...
29
votes
3answers
43k views

What is the difference in what AIC and c-statistic (AUC) actually measure for model fit?

Akaike Information Criterion (AIC) and the c-statistic (area under ROC curve) are two measures of model fit for logistic regression. I am having trouble explaining what is going on when the results of ...
14
votes
3answers
7k views

How to derive the probabilistic interpretation of the AUC?

Why is the area under the ROC curve the probability that a classifier will rank a randomly chosen "positive" instance (from the retrieved predictions) higher than a randomly chosen "positive" one (...
6
votes
1answer
999 views

How is a ROCAUC=1.0 possible with imperfect accuracy? [duplicate]

I used sklearn to compute roc_auc_score for a dataset of 72 instances. The accuracy was at 97% (2 misclassifications), but the ROC AUC score was 1.0. How is this ...
23
votes
1answer
3k views

Did I just invent a Bayesian method for analysis of ROC curves?

Preamble This is a long post. If you're re-reading this, please note that I've revised the question portion, though the background material remains the same. Additionally, I believe that I've devised ...
16
votes
3answers
14k views

Area under the ROC curve or area under the PR curve for imbalanced data?

I have some doubts about which performance measure to use, area under the ROC curve (TPR as a function of FPR) or area under the precision-recall curve (precision as a function of recall). My data is ...
8
votes
3answers
4k views

Choosing a classification performance metric for model selection, feature selection, and publication

I have a small, unbalanced data set (70 positive, 30 negative), and I have been playing around with model selection for SVM parameters using BAC (balanced accuracy) and AUC (area under the curve). I ...
5
votes
2answers
10k views

How to interpret 95% confidence interval for Area Under Curve of ROC?

(I am following this paper, from page 47 on http://www.bundesbank.de/Redaktion/EN/Downloads/Tasks/Banking_supervision/working_paper_no_14_studies_on_the_validation_of_internal_rating_systems.pdf?...
2
votes
1answer
995 views

Can someone sort me out regarding the calculation of AUC?

I am having some trouble with two different implementations of a classification problem giving different results. Me and my college who did the other implementation has narrowed the problem down to ...
8
votes
3answers
12k views

AUC and class imbalance in training/test dataset

I just start to learn the Area under the ROC curve (AUC). I am told that AUC is not reflected by data imbalance. I think it means that AUC is insensitive to imbalance in test data, rather than ...
2
votes
1answer
9k views

Sample size calculation for ROC/AUC analysis

As a background, I am not familiar with stats except on a basic level. I have been tasked with doing some analysis that is out of my comfort zone. I am trying to figure out how to compute necessary ...
27
votes
3answers
28k views

ROC curve for discrete classifiers like SVM: Why do we still call it a “curve”?, Isn't it just a “point”?

In the discussion : how to generate a roc curve for binary classification, I think that the confusion was that a "binary classifier" (which is any classifier that separates 2 classes) was for Yang ...
14
votes
1answer
3k views

Why is ROC AUC equivalent to the probability that two randomly-selected samples are correctly ranked? [duplicate]

I found there are two ways to understand what AUC stands for but I couldn't get why these two interpretations are equivalent mathematically. In the first interpretation, AUC is the area under the ...
13
votes
2answers
7k views

optimizing auc vs logloss in binary classification problems

I am performing a binary classification task where the outcome probability is fair low (aroung 3%). I am trying to decide whether to optimize by AUC or log-loss. As much as I have understood, AUC ...
14
votes
2answers
7k views

Evaluate Random Forest: OOB vs CV

When we assess the quality of a Random Forest, for example using AUC, is it more appropriate to compute these quantities over the Out of Bag Samples or over the hold out set of cross validation? I ...
8
votes
2answers
3k views

What to do AFTER nested cross-validation?

I've searched exhaustively on this forum and elsewhere, and have come across a lot of great material. However, I'm ultimately still confused. Here's a basic, concrete example of what I'd like to ...
18
votes
3answers
19k views

Can AUC-ROC be between 0-0.5?

Can AUC-ROC values be between 0-0.5? Does the model ever output values between 0 and 0.5?
6
votes
1answer
1k views

Is AUC via CV a good procedure for selecting optimal model?

I'm fitting a logit classifier with LASSO and cross-validation, and struggling to select the optimal model using AUC -instead of the more usual loss like binomial deviance or classification error. I ...
2
votes
1answer
1k views

ROC/AUC Confidence Interval

For a single ROC curve (with relevant AUC score), how can you calculate the confidence interval? (The data used to generate this ROC/AUC is available) Given my relatively limited background in this ...
5
votes
1answer
527 views

Evaluating models by Loglogss, AUC, and Accuracy

I am evaluating three models (say, A, B, and C) by three different metrics: Log-loss, AUC, and Accuracy. The results show that Log-loss: C>A>B (B has the best performance in terms of Log-loss) ...
5
votes
1answer
2k views

How is the optimal probability cut-off in a ROC defined by the R package Epi?

The plot below was created by R package Epi::ROC for a binary classification problem. There is an optimal cut off or threshold for the classification probability at ...
2
votes
1answer
90 views

When is accuracy score preferred to AUCROC?

I have a binary and balanced dataset. Do I have to see the AUROC as the different trade-offs between the TPR and the FPR and the accuracy as a result with a threshold of 0.5? When is accuracy a ...
33
votes
2answers
36k views

Area under Precision-Recall Curve (AUC of PR-curve) and Average Precision (AP)

Is Average Precision (AP) the Area under Precision-Recall Curve (AUC of PR-curve) ? EDIT: here is some comment about difference in PR AUC and AP. The AUC is obtained by trapezoidal interpolation ...
14
votes
3answers
16k views

What are the differences between AUC and F1-score?

F1-score is the harmonic mean of precision and recall. The y-axis of recall is true positive rate (which is also recall). So, sometime classifiers can have low recall but very high AUC, what that ...
8
votes
1answer
2k views

Optimizing for AUC

AUC is a popular classification evaluation metric. This is a measure of aggregate performance—do any of the standard loss functions (functions of an individual example's label & prediction) ...
7
votes
2answers
11k views

What is the formula to calculate the area under the ROC curve from a contingency table?

For example, if my table is: ...
6
votes
1answer
3k views

Statistics for Area under the ROC curve

I have a question regarding statistical evaluation of the AUC. In their paper (http://www.jstor.org/stable/2531595), DeLong et al. describe a method to evaluate AUC curves. (Another good explanation ...
5
votes
2answers
11k views

How to improve F1 score with skewed classes?

I've a dataset of roughly 40K samples, with 39.6K samples belonging to the target class 0 and 400 to class 1. I've tried several classification algorithms, without too much fine tuning, just to get a ...
13
votes
1answer
330 views

Does a logistic regression maximizing likelihood necessarily also maximize AUC over linear models?

Given a data set with binary outcomes $y\in\{0,1\}^n$ and some predictor matrix $X\in\mathbb{R}^{n\times p}$, the standard logistic regression model estimates coefficients $\beta_{MLE}$ which maximize ...
7
votes
1answer
8k views

What is AUC of PR-curve?

I understand that AUC under ROC curve is a classic evaluation measurement for classifiers (which is basically the accuracy). However, when data is imbalanced, PR will be alternative. So, what does the ...
6
votes
3answers
1k views

pattern of ROC curve and choice of AUC

I am using ROC curves and full AUC values to compare different models, using simulated data. Now I think I am confused with the interpretations of ROC curves and AUC values. Please see the figure ...
5
votes
1answer
4k views

Relationship between AUC and U Mann-Whitney statistic

Recently I learned about the relationship between Area Under (ROC) Curve and $U$ statistic of the Wilcoxon-Mann-Whitney test. It is supposed to follow the following rule (got it from this nice post on ...
3
votes
3answers
2k views

Can AUC decrease with additional variables?

I'm fitting a logistic regression model to predict probabilities from a set of variables. I'm comparing two such models, say M1 and ...
2
votes
1answer
417 views

Statistical Power of ROC/AUC Test with non-IID Samples :: To how many IID Samples are my non-IID Samples Equivalent?

I've been assigned to solve the following problem as part of a serious, biological research project. I think I have a tentative solution, but I'm wondering whether the approach I've picked is the best....
2
votes
2answers
1k views

Performance Metrics for Imbalanced Classification

I'm trying to fit multiple Stochastic Gradient Descent models to a dataset where the target (binary target, 0 or 1) is very imbalanced, i.e the success rate is about 0.0001. Out of all the models I'...
3
votes
1answer
4k views

Differences in AUC calculation in R between pROC and AUC

I was comparing the performance of pROC and AUC libraries when performing auc() calculations on random data: ...
2
votes
2answers
1k views

The distribution of the AUC

I am wondering how the confidence interval for the Area under the Curve statistic (ROC curves) is derived. I have heard that the AUC can be assumed to be normally distributed, but I am looking for a ...
2
votes
3answers
881 views

AUC with incomplete ROC curve

I am doing an experiments where changing a parameter I am obtaining different number of FalsePositive, FalseNegative... and so on. I am using this parameter tuning as threshold tuning to obtain FPR ...
1
vote
1answer
1k views

Differences between cross validation and bootstrapping to estimate the standard error of the AUC of a given ROC curve

I know there's been some discussion on differences between CV and bootstrapping for estimating out-of-sample prediction error of a classifier. For example, in here (Differences between cross ...
1
vote
1answer
3k views

How to distinguish overfitting and underfitting from the ROC AUC curve?

For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model. But how to distinguish whether a ...
0
votes
2answers
126 views

Statistical evidence that the AUC was not overfitted to the model. With N=119, C-stat = 0.81 seems optimistic. Optimism-adjusted?

My data have 119 cases and we did ROC for x (continuous variable) to predict postoperative y (categorical variable) available here, we got a comment from a reviewer asking: Please provide ...
0
votes
2answers
3k views

AUC vs accuracy for model accuracy evaluation [duplicate]

In R I tried to measure the accuracy by performing a classification analysis using logistic regression analysis. I found that there are two ways to measure accuracy. One is AUC measurement using ROC ...
4
votes
2answers
631 views

Appropriate way to get Cross Validated AUC

I was thinking about cross-validation and how it is the most appropriate way to do it... Let's take the case of binary logistic regression where the goal is to calculate the AUC. Make the partition ...
3
votes
1answer
502 views

AUC and Its Usage

I recently learned AUC and ROC and confused about the usage of AUC. What if my model A has a higher AUC than my model B but in terms of one particular threshold--one particular point on the ROC graph--...