Skip to main content

# Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operating characteristic (ROC) curve.

623 questions
Filter by
Sorted by
Tagged with
3 votes
1 answer
110 views

### Why would pROC::roc calculate $\max\{AUC, 1 - AUC\}$ by default?

There is some interesting behavior in the pROC::roc function in R. ...
• 64.8k
1 vote
0 answers
87 views

### AUC > 0.5 under null model following feature selection

I've been going over the output of a Monte Carlo model that simulates disease risk as a function of genotype. Under a null model of no disease risk, we have 1000 case and 1000 control individuals. ...
• 133
0 votes
0 answers
13 views

### (How) can ROC/AUC analysis be adapted to problems with two (or more) thresholds?

(How) can ROC/AUC analysis be adapted to problems with two (or more) thresholds? Example 1: We measure variable $x$, assign label $1$, if it falls into interval $[a, b]$, and label $0$ otherwise. We ...
• 4,429
0 votes
0 answers
13 views

### pROC plot shape with only one point

I've done this plot whit pROC ...
• 213
0 votes
1 answer
54 views

### How to interpret this confusion matrix in R?

I've produced a logistic regression model using 70% of my data and tested it using the remaining 30%. These are the results of my confusion matrix: Confusion Matrix and Statistics Confusion Matrix (...
• 21
0 votes
1 answer
18 views

### AUC comparison with Delong when i.i.d is broken due to clustered data

I am asked to compare two AUC ROCs and output confidence interval for both. Delong method allows (and also Fast Delong) to have a stronger test than usual bootstrapping method. The data I am working ...
0 votes
0 answers
37 views

### Compare bootstrap auc confidence interval using t-test

In order to choose between a machine learning model when the number of features is 5 and a machine learning model when the number of features is 6, I want to bootstrap the auc of the model to obtain a ...
• 89
0 votes
0 answers
20 views

### Is DeLong test valid for comparing two independent AUCs?

I would like to know if DeLong test valid for comparing two independent AUCs? or it is only for two dependent AUCs?
• 129
0 votes
0 answers
15 views

### Consequences of maintaining IID assumption for prediction model training, but relaxing it for model testing

Let's say you're developing a prediction model, and you are confident that your data are IID. For example, you have a dataset where each row represents a different patient, and you build a model to ...
0 votes
0 answers
9 views

### Selection of important features through cross validation and shape value importance

To extract important features for the binary classification problem, recursive feature elimniation was performed based on the importance value of the shap value through nested cv. The first thing I am ...
• 89
1 vote
1 answer
72 views

### How to evaluate multi-class classifier on probability prediction task?

I have a balanced dataset where each object (song) has one of the four target class labels (mood of a song). Example: ID feature1 feture2 feature3 target_class 0 0.5 0.11 125 upbeat 1 0.23 0.75 136 ...
0 votes
0 answers
21 views

### Why would area under the PR curve include points off of the Pareto front?

(Let's set aside thoughts about if we should be calculating PR curves or areas under them at all.) A precision-recall curve for a "classification" model can contain points that should not be ...
• 64.8k
2 votes
1 answer
107 views

### Comparing probability threshold graphs for F1 score for different models

Below are two plots, side-by side, for an imbalanced dataset. We have a very large imbalanced dataset that we are processing/transforming in different manner. After each transformation, we run an ...
0 votes
0 answers
27 views

### How to calculate AUC for a P-R curve with unusual starting point

I am working with a binary classifier that is outputting scores between 0 and 1, indicating probabilities of class membership, according to the model. I produced a P-R curve and the first point (i.e., ...
• 187
2 votes
1 answer
28 views

### Relating population-level AUC (Somers's $D_{xy}$) to a mean shift

Say we have group $0$ distributed as $N(\mu, \sigma^2)$ and group $1$ distributed as $N(\mu+\delta, \sigma^2)$. We then use the Gaussian-distributed variables to predict group membership. It seems ...
• 64.8k
0 votes
0 answers
29 views

### Valid confidence intervals for AUC with very few cases?

Situation: I have a data set of patients who are positive for a certain illness. I also have a score for 1-year mortality from this illness, designed using a separate data set. The plan is to check ...
• 213
0 votes
0 answers
50 views

### Relation between gini coefficient/accuracy ratio and roc_auc_score when there are many identical predictions

I have been working on ranking metrics related to various estimators lately, and cam a across a curious phenomenon related to the Gini-coefficient which I would like to understand better. I will start ...
0 votes
0 answers
23 views

### General threshold-independent statistics

apologies in advance for my bad statistics. I am trying to evaluate two models based on their adverse impact ratio (AIR), which is defined as  AIR = \frac{\text{approval rate for protected class}}{\...
• 113
6 votes
1 answer
76 views

### ROCAUC = average sensitivity across all thresholds according to IEEE TPAMI, yet my calculations show otherwise

Carrington et al (2023) make the claim that area under the receiver-operator characteristic curve is equal to the average sensitivity across all thresholds, and similarly for specificity (section 3), ...
• 64.8k
1 vote
1 answer
97 views

### Integrated biomarkers for ROC

I'm am looking for ways to combine multiple quantitative values in order to build a ROC curve with specificity and sensitivity. This seems to be common in multiple biomarkers paper, but I can't find ...
• 13
0 votes
0 answers
21 views

### Can the Log of PR AUC curve plot be any useful?

I was doing some tests regarding my PR curve for 2 different models (first image), and I got the idea of ploting the log of those curves (second image) to see if there were any insights that I could ...
2 votes
0 answers
56 views

### C-statistic and measuring the contextual effect in multilevel logistic regression

I have a two-level logistic regression model where the outcome is "InfectedqPCR" (Plasmodium-infected as determined by qPCR) at the individual level. I have a range of individual- and ...
• 171
5 votes
1 answer
277 views

### Is ROC curve unique?

ROC curve and the area under it (AUC) are routinely used to evaluate the performance of binary classifiers. However, it seems that both, the shape of the curve and the area, depend on the parameter ...
• 4,429
0 votes
0 answers
14 views

### What maximum value of AUC optimism could still be allowed to confirm that logistic regression model does not overfit?

I am not sure how to define that a statistical model does not overfit based on a difference between bootstrapped AUC and AUC calculated on all training data. In the literature I saw 2 approches. The ...
• 101
0 votes
0 answers
11 views

### Is it meaningful to compare within-group AUROC between groups?

I have a risk model that I want to evaluate on different (patient) groups in order to compare how well the model is working on each of them. The groups may differ in size, baseline / prevalence / ...
• 3,118
1 vote
1 answer
135 views

### Should we use train, validation, or test data when creating PR/AUC curves to optimize the decision threshold?

It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes ...
1 vote
1 answer
30 views

### Is it preferred to evaluate with a metric at a single decision threshold (eg Fbeta) vs averageing across thresholds (eg ROC-AUC)

Consider these two approaches to evaluating a classifiers performance: Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be ...
1 vote
0 answers
63 views

### Am I able to compare pooled AUC values (from a meta-analysis) of two similar tests conducted in the same samples to establish which may be superior? [closed]

I'm new to all of this so I apologise in advance. I'm currently conducting a meta-analysis and have pooled AUCs that represent the accuracy of two very similar prognostic factors in predicting ...
• 11
3 votes
2 answers
163 views

### Calculate area under precision-recall curve from area under ROC curve and the prevalence

I am reading material that reports the area under a ROC curve. I am curious to know what the performance would be in precision-recall space. From the sensitivity and specificity values in the ROC ...
• 64.8k
2 votes
1 answer
344 views

### Confidence Intervals of ROC Curve's AUCs overlap but delong test is significant?

I am using ci.auc in the pROC library to calculate AUC's confidence intervals and roc.test to calculate delong test. When I run the following: ...
• 21
1 vote
0 answers
14 views

### RecSys model performance stalling at 47% AUC and F1-Score. Is the problem due to ratio of users to items in my dataset?

I'm having trouble with making my validation metrics go down for the binary_crossentropy and go up for the F1-score and AUC. I've tried tuning my hyper parameters such as the number of latent features ...
2 votes
1 answer
1k views

### Delong's test for comparing the significance difference of two AUC

I have done two prediction model in R as an example as below: ...
• 129
0 votes
0 answers
25 views

### De Longs test on ROC curves?

I have two different ROC curves for Model A and B. I wanted to calculate the DeLongs test to identify the statisical significance of the difference between the model. model_A_tpr = [0, 0.2, 0.4, 0.6, ...
• 101
-1 votes
1 answer
120 views

### difference between c-statistics and AUC for prediction performance

I have two types of outcome from a big dataset and I performed prediction models and checked the prediction performance using 10-fold cross-validation, so: for binary outcome: lasso logistic ...
• 129
2 votes
3 answers
266 views

### cutoff and auc and changing cutoff

can you tell me if this is ok? While the AUC (i.e. AUC of 0.6) we got is acceptable since it's bigger than 0.5, we may need to re-evaluate at our cutoff selections again. Because we can select cutoffs ...
0 votes
0 answers
52 views

### C-statistic (or AUC) for fractional logistic regression (i.e. continuous regression)

I have proportional data to which I have fit a logistic regression (i.e. fractional logistic regression). The statistician in our group wants me to provide a c-statistic for the regression. My ...
• 356
26 votes
6 answers
1k views

### Area under the ROC curve when there is imbalance: is there a problem, and if not, why does this rumor exist?

THE BOUNTY As promised, a bounty of $250$ points has been issued. A bounty-worthy answer should address the apparent controversy in the answers here that ROC curve interpretation does not depend on ...
• 64.8k
0 votes
0 answers
10 views

### Use of area under the curve to compare treatments with count ordinal data over time

I am working on questionnaire data, looking into bimonthly responses where "improvement" and "worsening" for different health parameters is recorded as counts or frequencies for a ...
2 votes
0 answers
961 views

### C-statistic vs AUC [closed]

I am analysing diagnostic accuracy. I have a dataset with a ground truth and 3 predictors. Ground truth = binary (0/1) Predictor 1-2 = binary (0/1) Predictor 3 = continuous (0-100) I have 50,000 ...
• 139
0 votes
0 answers
19 views

### Question regarding deriving the AUC value

I have a dataset (auc_data) which includes the response variable (y) and the predicted probabilities (y_hat). I have calculated the AUC value based on this data. You can see that there are 3 rows in ...
• 889
2 votes
0 answers
277 views

### Power/sample size estimation when comparing two AUCs (area under the curve)?

Are there any R functions (or other free software) for calculating power/sample size needed to compare 2 AUCs (area under the curve)? Specifically, suppose you want to fit 2 different models to a ...
• 21
1 vote
1 answer
369 views

### Why use average_precision_score from sklearn? [duplicate]

I have precision and recall values and want to measure an estimator performance: ...
• 31
0 votes
0 answers
13 views

### Can correlating features change the AUROC of a model trained on randomly shuffled labels?

I did a binary classification with a random forest on my dataset, where I removed all correlating features and figured I'd try out a sort of "negative control" and shuffle all labels ...
• 1
1 vote
1 answer
140 views

### Confused about AUC of 0 from a logistic regression classifier with a non-informative predictor

I am using logistic regression as a classifier to predict the variable Group (old/young) from a variable called Sentence_length. When I plot density functions of Sentence_length by Group, it is clear ...
• 197
0 votes
1 answer
32 views

### Interpretation of ROC curve curving early

I am running a binary LASSO logistic regression using glmnet. The initial data I work with is raster spatial data. When I create an ROC (AUC ~ 0.72) curve based on the test data, the resulting curve ...
2 votes
1 answer
248 views

### ROC-AUC in GEE models?

I am wondering if it is possible to make ROC-AUC curves for GEE models? I found few papers who did that and it wasn't clear for me. I thought it was impossible given how they are marginal models. ...
1 vote
0 answers
72 views

### Combination of AUC for different subset of samples

Assuming I have ROC-AUC for multiple subsets of samples, is there a way to produce a "weighted" average of AUC or something that will represent/approximate the AUC for the complete ...
• 390
1 vote
0 answers
290 views

### Interpretation of area under the precision-recall curve

The area under the receiver-operator characteristic curve has a interpretation of how well the predictions of two categories are separated. This post gives the area under the precision-recall curve as ...
• 64.8k
11 votes
6 answers
3k views

### Where in the ROC curve does it tell you what the threshold is?

In my understanding, the ROC curve plots the True positive rate and the False positive rate. However, I've also read in other places that the ROC curve helps determine where the threshold for ...
• 1,011
1 vote
0 answers
103 views

### Using the Bootstrap in conjunction with Cross Validation for Confidence Intervals

I am trying to estimate confidence bounds for prediction metrics of a binary-outcome logistic regression model. In particular, the AUC and the Brier score. I've looked at many other posts on this ...