# Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.

542 questions
Filter by
Sorted by
Tagged with
11 views

### Why PR score is down when balanced accuracy is good?

I just read this discussion here and here. I have a dataset of 977 records where class proportion is 77:23. My balanced accuracy is 75.5, ...
• 1,758
38 views

### How is AUC helpful when we only need one threshold of a classifier

AUC is a summation of performance at different thresholds, but do we only care about a good performance at one threshold? Imagine a classifier with a low ROC but shots up at point of a low FP and high ...
• 21
15 views

### Understand AUCs of different models

I'm testing two models against each other: provides an AUC of 93,94 % with TP = 99,9 %, TN = 0 %, FP = 100 % and FN = 0.1 %. provides an AUC of 92,78 % with TP = 98,8 %, a TN = 30,6 %, FP = 69,4 % ...
• 2,187
156 views

### If the AUC score is 100 percent can F1 value be 99.94 percent?

If the AUC score is 100 percent can the F1 value be 99.94 percent? I would expect 100 percent, too.
• 189
9 views

### Why R survAUC::AUC functions require both training and test data?

I am familiar with binary classification models, where the AUC is the metric from the area under the specificity-sensitivity curve, it indicates the performance on a dataset. Now, I'd like to assess ...
• 21
23 views

### Make ROC graph with my problematic [duplicate]

For a personal project, i'm trying to figure out how to trace some ROC/AUC graph with my current problematic. I have a list of thunder flashes, and i'm trying to find if a meteorological variable (...
13 views

### Deriving Bayesian Credible Intervals for AUC using R brms

I am trying to estimate the posterior distribution for the AUC of a predictive biomarker using R brms. However, whenever I calculate the AUC using the posterior distribution of the model parameters, ...
• 43
23 views

### How to compare multiple survival scores regarding its accuracy?

I have a clinical dataset which has already common prognosis scores, like characteristic tumor markers, tumor classification, etc. However I developed a new score which is able to predict patients' ...
16 views

### How to interpret this confusion matrix and roc curve?

I got these two images for evaluating a RF: I wonder why the ROC curve seems to be so good while the confusion matrix shows that the True Positive isn't so good with only ~16 %? By looking only at ...
• 2,187
14 views

### AUC first drops before starting to grow

I have a very imbalanced classification problem, 99% vs 1%, and I am using logistic regression in pytorch. I often see the initial weights, randomly initialized, achieve AUC of 60-70% before any ...
1 vote
22 views

### Can I use the AUR under the ROC on unbalanced test data?

I have split my data into training and test data, built several prediction models and now I want to evaluate the models using the test data set.The data is very unbalanced so I balanced the training ...
• 11
49 views

### AUC - Logistic Regression versus LDA, and Naive Bayes

everyone! I am a newbie on machine learning, and I am now interested on classification modeling. I used logistic regression, linear discriminant analysis (LDA), and naive Bayes on my notebook DataCamp ...
121 views

### Appropriate way to get cross validated performance metrics

For cross-validation of a logistic regression classifier, it seems to me that there are a number of different approaches to calculating each performance metric: The performance metric is calculated ...
59 views

### Difference between ROC-AUC and Multiclass AUC (MAUC)

I am trying to understand the interpretation of these metrics in a multiclass scenario: ROC-AUC and MAUC. Scikit-learn provides ...
• 231
66 views

### Comparing the difference in test AUC between different machine learning models using Delong's test

I'm in the process of developing four machine learning models (Neural network, logistic regression, XGBoost, SVM) on the same training data using the same exact features (10 features in my model) and ...
1 vote
52 views

### AUC between estimated lines

Let's say you have a lmer model that test the drug effect of a set of rats with a set of rats (Control): lme1 <- lmer(lVolume ~ Days*Drug + (Days|Drug)) where ...
• 111
23 views

### Can anyone explain how to calculate AUC_DS, AUC_BW, and AUC_0 mentioned in this image?

I know AUC is for binary classification, but in a paper, the authors seemingly used AUC_DS, AUC_BW, and AUC_0 to compare vector values, like ground truth of one sample [3,2,1,0,1,2,3] prediction value ...
• 1
26 views

### My data can be approximated with normal distribution (multimodal). How can I find the reasons and explain this behaviour?

I use DeLonge method to compare two ROC AUCS. The result of it is Z-score. Both ROC AUCs obtained from LDA (linear discriminant analysis) from sklearn package. The ...
69 views

### Precision-Recall curve and AUC in multi-class problems using R

I'm trying to evaluate a RandomForest model for multi-class classification using the Area Under Precision-Recall Curve. I need to plot the PR curve of each class and the micro and macro average and ...
• 113
66 views

### How to report AUC from Cox Regression?

I need to calculate the AUC for a Cox regression model. SAS is giving me a time dependent AUC as below. How do I report this for the paper? Thanks!!
• 55
23 views

### Feature Selection + Unpenalized Logistic Regression AUC

I am building a predictive model using a highly-dimensional dataset. To reduce dimensionality and reduce overfitting, I used LASSO regression to select features. ...
20 views

### Ridge regression coefficients show model importance but the model evaulation not

I have performed two ridge logistic regressions in R to check which of the two models perform better. From the first look of the coefficients, it looks like model1 ...
• 332
78 views

### If going with the opposite prediction of a bad predictor gives good predictions, why not do that?

Let’s restrict our consideration to binary outcomes. I have a friend who is terrible at predicting the future, always predicting the opposite of what winds up happening. For instance, my friend ...
• 31k
31 views

### How to compute ROC AUC for a method that uses two models?

To compare my method with others I'm trying to compute its AUC but I got a bit confused on how to do this for my case. My method uses a model that classifies an image as class A or B, after that if ...
1 vote
74 views

### Proper way to finding area under ROC curve (AUC)

I have data of 200 persons with 2 variables: 1. C_level: Blood level of a chemical C (a numeric value) 2. Disease: Yes or No I want to know if C_level can be used ...
• 8,993
553 views

### Validity of AUC for binary categorical variables

Scikit-learn function roc_auc_score can be used to get area under curve (AUC) of ROC curve. This score is generally used for numeric predictors' value in predicting ...
• 8,993
40 views

### Is it valid to do ROC analysis without using test data?

I am trying to do an ROC analysis to check whether a particular biomarker has any diagnostic power for screening a condition. Since I have only small dataset of 100. I am not splitting the data into ...
• 811
17 views

### Can you check the significance of your AUC values? [duplicate]

I have computed some AUC values from the ROC curve based on logistic regressions. Firstly, I have divided my two datasets (D1, D2...
• 332
30 views

### ROC AUC for average and geometric mean of two models

Suppose I have two binary classification models built on the same dataset. One has ROC AUC of 0.86, another one has ROC AUC of 0.92. What will be the ROC AUC for the average of two models and their ...
81 views

### How to simulate a calibrated prediction model given prevalence and auc

I want to be able to simulate a prediction model given some prevalence of the event and the AUC of the model. I followed the method proposed here but, although this works for giving AUC and predicted ...
17 views

### AUC Instability when averaging over individual readers

I am trying to evaluated an algorithm that supports doctors when making a diagnosis. I have recruited 10 doctors. I have 50 training examples. Each doctor is randomly assigned 25 cases to review alone ...
16 views

### Does the imbalance in my data affect model quality in this case?

I have two datasets, used for classification task: training and validation one. Both of them are equally imbalanced, with about 24% of target value equal 1 and 75% of target value equal 0. I am using ...
• 113
41 views

### AUC and misclassification rate [duplicate]

why do you think that is possible? I compare decision tree with neural network in SAS EM and it chooses DT as the best model, based on misclassification rate, I guess. But AUC is higher for neural ...
1 vote
69 views

### PRAUC curve Interpretation on imbalanced data

My training data is undersampled to a positive to negative ratio of 5%. I observe a PRAUC of .4 on my training data. When I test the model on real-world data where the positive to negative ratio is .5%...
• 21
20 views

### Model selection in presence of overfitting - better test or closer train

Suppose I have a tree-based model (Random Forest for the sake of the example) and I play with a regularization parameter (tree depth) to fight overfitting. Eventually I can come up with two models - ...
• 318
1 vote
143 views

### Training AUC below 0.5

I've trained a logistic regression using a small number of predictors - pseudo R-squared is only 0.1 but I have significant terms and a nice low p value for the model. However, even on its own ...
• 193
37 views

### The different usage of ROC between diagnotic tests and machine learning

I am currently very confused about the ROC usage in diagnostic tests and machine learning. In the scenario of medical diagnostic studies, many tutorial does not mention the data split procedure as ...
• 43
41 views

### Derivation of closed-form ROC expression under binormal assumptions

It's a known result that, under binormality assumptions, the area under the ROC curve (AUC) for a binary classifier has the following closed form. Formally, define the class conditional mean and ...
• 1,066
8 views

### Is it possible to build an imputed 2-variable AUC model

I am building a 2-variable predictive model but there are some missing data for both of my predictors. Is it possible to combine MICE code with AUC code to get something called "imputed AUC model&...
• 311
1 vote
29 views

### How to choose the balance between model fit vs AUC (diagnostic accuracy)?

I would like to know how we can choose between model fit (calibration) vs AUC when building the predictive model. For example, if I have one predictor which improves the model fit but results in a ...
• 311
1 vote
47 views

### Relation between AUROC and threshold

As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that ...
• 481
197 views

### AUC with different threshold

I know AUC is supposed to be independent on the threshold, which means AUC does not change while the threshold changes. However, I'm getting different AUC values while changing the thresholds. I'm ...
• 399
55 views

### Can I conclude that the classifier is always good when Precision-Recall Curve above the baseline?

I used logistic regression for highly imbalaned data (1=0.6% , 0=99.4%) Since PR curves are sensitive to imbalance, so i used it, but I don't know how to interpret graph appropriately. This is PR-...
108 views

### how to caculate 95% CI for AUC? try 384 times or Hanley et al. (1982) method?

I am working on a prediction task to predict heart disease risk. The data size is around 1500 and is splitted into train, validate and test datasets. I am use train dataset to train and use validate ...
41 views

### How to calculate chance-level f1, ROC-AUC, PR-AUC for imbalanced dataset

I have an imbalance dataset (60% class 1, 40% class 0). I trained a model and got accuracy, f1, ROC-AUC and PR-AUC. I want to compare them to chance-level performance. obviously chance-level of acc if ...
• 27
1 vote
86 views

### Given scores for a classifier and an expected positive rate, generate targets that achieve a given AUC score

Say I have output scores (maybe they're logits or poorly calibrated probabilities) from a trained classifier like y_hat = [1, 2, 3, 4], an expected positive rate of 25%, and I want to generate targets ...
• 1,172
191 views

### Build ROC curve with only one class

I have three datasets structured as well: ...
• 115
1 vote
273 views

### Why would AUC on a validation set increase while loss increases?

I'm training a deep learning model in PyTorch. The first two images I posted here make perfect sense as they are the classical idea of overfitting. The training loss keeps decreasing while the ...
• 1,525
I need to calculate the area under curve (AUC) with SD for several repeats of an experiment. I am using the trapeze method to estimate the area, where $$S = \sum(S_i)$$ and  S_i = (a+b)*d/2 = (f(...