Skip to main content

Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operating characteristic (ROC) curve.

Filter by
Sorted by
Tagged with
3 votes
1 answer
110 views

Why would `pROC::roc` calculate $\max\{AUC, 1 - AUC\}$ by default?

There is some interesting behavior in the pROC::roc function in R. ...
Dave's user avatar
  • 64.8k
1 vote
0 answers
87 views

AUC > 0.5 under null model following feature selection

I've been going over the output of a Monte Carlo model that simulates disease risk as a function of genotype. Under a null model of no disease risk, we have 1000 case and 1000 control individuals. ...
Max's user avatar
  • 133
0 votes
0 answers
13 views

(How) can ROC/AUC analysis be adapted to problems with two (or more) thresholds?

(How) can ROC/AUC analysis be adapted to problems with two (or more) thresholds? Example 1: We measure variable $x$, assign label $1$, if it falls into interval $[a, b]$, and label $0$ otherwise. We ...
Roger V.'s user avatar
  • 4,429
0 votes
0 answers
13 views

pROC plot shape with only one point

I've done this plot whit pROC ...
ArTu's user avatar
  • 213
0 votes
1 answer
54 views

How to interpret this confusion matrix in R?

I've produced a logistic regression model using 70% of my data and tested it using the remaining 30%. These are the results of my confusion matrix: Confusion Matrix and Statistics Confusion Matrix (...
Laura's user avatar
  • 21
0 votes
1 answer
18 views

AUC comparison with Delong when i.i.d is broken due to clustered data

I am asked to compare two AUC ROCs and output confidence interval for both. Delong method allows (and also Fast Delong) to have a stronger test than usual bootstrapping method. The data I am working ...
Tamsina Ludwig's user avatar
0 votes
0 answers
37 views

Compare bootstrap auc confidence interval using t-test

In order to choose between a machine learning model when the number of features is 5 and a machine learning model when the number of features is 6, I want to bootstrap the auc of the model to obtain a ...
JAE's user avatar
  • 89
0 votes
0 answers
20 views

Is DeLong test valid for comparing two independent AUCs?

I would like to know if DeLong test valid for comparing two independent AUCs? or it is only for two dependent AUCs?
user358238's user avatar
0 votes
0 answers
15 views

Consequences of maintaining IID assumption for prediction model training, but relaxing it for model testing

Let's say you're developing a prediction model, and you are confident that your data are IID. For example, you have a dataset where each row represents a different patient, and you build a model to ...
rbly004's user avatar
0 votes
0 answers
9 views

Selection of important features through cross validation and shape value importance

To extract important features for the binary classification problem, recursive feature elimniation was performed based on the importance value of the shap value through nested cv. The first thing I am ...
JAE's user avatar
  • 89
1 vote
1 answer
72 views

How to evaluate multi-class classifier on probability prediction task?

I have a balanced dataset where each object (song) has one of the four target class labels (mood of a song). Example: ID feature1 feture2 feature3 target_class 0 0.5 0.11 125 upbeat 1 0.23 0.75 136 ...
primadonna's user avatar
0 votes
0 answers
21 views

Why would area under the PR curve include points off of the Pareto front?

(Let's set aside thoughts about if we should be calculating PR curves or areas under them at all.) A precision-recall curve for a "classification" model can contain points that should not be ...
Dave's user avatar
  • 64.8k
2 votes
1 answer
107 views

Comparing probability threshold graphs for F1 score for different models

Below are two plots, side-by side, for an imbalanced dataset. We have a very large imbalanced dataset that we are processing/transforming in different manner. After each transformation, we run an ...
Ashok K Harnal's user avatar
0 votes
0 answers
27 views

How to calculate AUC for a P-R curve with unusual starting point

I am working with a binary classifier that is outputting scores between 0 and 1, indicating probabilities of class membership, according to the model. I produced a P-R curve and the first point (i.e., ...
CopyOfA's user avatar
  • 187
2 votes
1 answer
28 views

Relating population-level AUC (Somers's $D_{xy}$) to a mean shift

Say we have group $0$ distributed as $N(\mu, \sigma^2)$ and group $1$ distributed as $N(\mu+\delta, \sigma^2)$. We then use the Gaussian-distributed variables to predict group membership. It seems ...
Dave's user avatar
  • 64.8k
0 votes
0 answers
29 views

Valid confidence intervals for AUC with very few cases?

Situation: I have a data set of patients who are positive for a certain illness. I also have a score for 1-year mortality from this illness, designed using a separate data set. The plan is to check ...
zxmkn's user avatar
  • 213
0 votes
0 answers
50 views

Relation between gini coefficient/accuracy ratio and roc_auc_score when there are many identical predictions

I have been working on ranking metrics related to various estimators lately, and cam a across a curious phenomenon related to the Gini-coefficient which I would like to understand better. I will start ...
user405288's user avatar
0 votes
0 answers
23 views

General threshold-independent statistics

apologies in advance for my bad statistics. I am trying to evaluate two models based on their adverse impact ratio (AIR), which is defined as $$ AIR = \frac{\text{approval rate for protected class}}{\...
stressed's user avatar
  • 113
6 votes
1 answer
76 views

ROCAUC = average sensitivity across all thresholds according to IEEE TPAMI, yet my calculations show otherwise

Carrington et al (2023) make the claim that area under the receiver-operator characteristic curve is equal to the average sensitivity across all thresholds, and similarly for specificity (section 3), ...
Dave's user avatar
  • 64.8k
1 vote
1 answer
97 views

Integrated biomarkers for ROC

I'm am looking for ways to combine multiple quantitative values in order to build a ROC curve with specificity and sensitivity. This seems to be common in multiple biomarkers paper, but I can't find ...
Renaud's user avatar
  • 13
0 votes
0 answers
21 views

Can the Log of PR AUC curve plot be any useful?

I was doing some tests regarding my PR curve for 2 different models (first image), and I got the idea of ploting the log of those curves (second image) to see if there were any insights that I could ...
GabrielPast's user avatar
2 votes
0 answers
56 views

C-statistic and measuring the contextual effect in multilevel logistic regression

I have a two-level logistic regression model where the outcome is "InfectedqPCR" (Plasmodium-infected as determined by qPCR) at the individual level. I have a range of individual- and ...
Trypanosoma's user avatar
5 votes
1 answer
277 views

Is ROC curve unique?

ROC curve and the area under it (AUC) are routinely used to evaluate the performance of binary classifiers. However, it seems that both, the shape of the curve and the area, depend on the parameter ...
Roger V.'s user avatar
  • 4,429
0 votes
0 answers
14 views

What maximum value of AUC optimism could still be allowed to confirm that logistic regression model does not overfit?

I am not sure how to define that a statistical model does not overfit based on a difference between bootstrapped AUC and AUC calculated on all training data. In the literature I saw 2 approches. The ...
zubenel's user avatar
  • 101
0 votes
0 answers
11 views

Is it meaningful to compare within-group AUROC between groups?

I have a risk model that I want to evaluate on different (patient) groups in order to compare how well the model is working on each of them. The groups may differ in size, baseline / prevalence / ...
Eike P.'s user avatar
  • 3,118
1 vote
1 answer
135 views

Should we use train, validation, or test data when creating PR/AUC curves to optimize the decision threshold?

It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes ...
another_student's user avatar
1 vote
1 answer
30 views

Is it preferred to evaluate with a metric at a single decision threshold (eg Fbeta) vs averageing across thresholds (eg ROC-AUC)

Consider these two approaches to evaluating a classifiers performance: Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be ...
another_student's user avatar
1 vote
0 answers
63 views

Am I able to compare pooled AUC values (from a meta-analysis) of two similar tests conducted in the same samples to establish which may be superior? [closed]

I'm new to all of this so I apologise in advance. I'm currently conducting a meta-analysis and have pooled AUCs that represent the accuracy of two very similar prognostic factors in predicting ...
Jim's user avatar
  • 11
3 votes
2 answers
163 views

Calculate area under precision-recall curve from area under ROC curve and the prevalence

I am reading material that reports the area under a ROC curve. I am curious to know what the performance would be in precision-recall space. From the sensitivity and specificity values in the ROC ...
Dave's user avatar
  • 64.8k
2 votes
1 answer
344 views

Confidence Intervals of ROC Curve's AUCs overlap but delong test is significant?

I am using ci.auc in the pROC library to calculate AUC's confidence intervals and roc.test to calculate delong test. When I run the following: ...
Eddie's user avatar
  • 21
1 vote
0 answers
14 views

RecSys model performance stalling at 47% AUC and F1-Score. Is the problem due to ratio of users to items in my dataset?

I'm having trouble with making my validation metrics go down for the binary_crossentropy and go up for the F1-score and AUC. I've tried tuning my hyper parameters such as the number of latent features ...
Mig Rivera Cueva's user avatar
2 votes
1 answer
1k views

Delong's test for comparing the significance difference of two AUC

I have done two prediction model in R as an example as below: ...
user358238's user avatar
0 votes
0 answers
25 views

De Longs test on ROC curves?

I have two different ROC curves for Model A and B. I wanted to calculate the DeLongs test to identify the statisical significance of the difference between the model. model_A_tpr = [0, 0.2, 0.4, 0.6, ...
a_parida's user avatar
  • 101
-1 votes
1 answer
120 views

difference between c-statistics and AUC for prediction performance

I have two types of outcome from a big dataset and I performed prediction models and checked the prediction performance using 10-fold cross-validation, so: for binary outcome: lasso logistic ...
user358238's user avatar
2 votes
3 answers
266 views

cutoff and auc and changing cutoff

can you tell me if this is ok? While the AUC (i.e. AUC of 0.6) we got is acceptable since it's bigger than 0.5, we may need to re-evaluate at our cutoff selections again. Because we can select cutoffs ...
Shawn Kim's user avatar
0 votes
0 answers
52 views

C-statistic (or AUC) for fractional logistic regression (i.e. continuous regression)

I have proportional data to which I have fit a logistic regression (i.e. fractional logistic regression). The statistician in our group wants me to provide a c-statistic for the regression. My ...
Brandon's user avatar
  • 356
26 votes
6 answers
1k views

Area under the ROC curve when there is imbalance: is there a problem, and if not, why does this rumor exist?

THE BOUNTY As promised, a bounty of $250$ points has been issued. A bounty-worthy answer should address the apparent controversy in the answers here that ROC curve interpretation does not depend on ...
Dave's user avatar
  • 64.8k
0 votes
0 answers
10 views

Use of area under the curve to compare treatments with count ordinal data over time

I am working on questionnaire data, looking into bimonthly responses where "improvement" and "worsening" for different health parameters is recorded as counts or frequencies for a ...
Colbie's user avatar
  • 1
2 votes
0 answers
961 views

C-statistic vs AUC [closed]

I am analysing diagnostic accuracy. I have a dataset with a ground truth and 3 predictors. Ground truth = binary (0/1) Predictor 1-2 = binary (0/1) Predictor 3 = continuous (0-100) I have 50,000 ...
user12541161's user avatar
0 votes
0 answers
19 views

Question regarding deriving the AUC value

I have a dataset (auc_data) which includes the response variable (y) and the predicted probabilities (y_hat). I have calculated the AUC value based on this data. You can see that there are 3 rows in ...
student_R123's user avatar
2 votes
0 answers
277 views

Power/sample size estimation when comparing two AUCs (area under the curve)?

Are there any R functions (or other free software) for calculating power/sample size needed to compare 2 AUCs (area under the curve)? Specifically, suppose you want to fit 2 different models to a ...
arbetj's user avatar
  • 21
1 vote
1 answer
369 views

Why use average_precision_score from sklearn? [duplicate]

I have precision and recall values and want to measure an estimator performance: ...
Ars ML's user avatar
  • 31
0 votes
0 answers
13 views

Can correlating features change the AUROC of a model trained on randomly shuffled labels?

I did a binary classification with a random forest on my dataset, where I removed all correlating features and figured I'd try out a sort of "negative control" and shuffle all labels ...
MWeb's user avatar
  • 1
1 vote
1 answer
140 views

Confused about AUC of 0 from a logistic regression classifier with a non-informative predictor

I am using logistic regression as a classifier to predict the variable Group (old/young) from a variable called Sentence_length. When I plot density functions of Sentence_length by Group, it is clear ...
Galit's user avatar
  • 197
0 votes
1 answer
32 views

Interpretation of ROC curve curving early

I am running a binary LASSO logistic regression using glmnet. The initial data I work with is raster spatial data. When I create an ROC (AUC ~ 0.72) curve based on the test data, the resulting curve ...
user avatar
2 votes
1 answer
248 views

ROC-AUC in GEE models?

I am wondering if it is possible to make ROC-AUC curves for GEE models? I found few papers who did that and it wasn't clear for me. I thought it was impossible given how they are marginal models. ...
Youknowme's user avatar
1 vote
0 answers
72 views

Combination of AUC for different subset of samples

Assuming I have ROC-AUC for multiple subsets of samples, is there a way to produce a "weighted" average of AUC or something that will represent/approximate the AUC for the complete ...
Gideon Kogan's user avatar
1 vote
0 answers
290 views

Interpretation of area under the precision-recall curve

The area under the receiver-operator characteristic curve has a interpretation of how well the predictions of two categories are separated. This post gives the area under the precision-recall curve as ...
Dave's user avatar
  • 64.8k
11 votes
6 answers
3k views

Where in the ROC curve does it tell you what the threshold is?

In my understanding, the ROC curve plots the True positive rate and the False positive rate. However, I've also read in other places that the ROC curve helps determine where the threshold for ...
Katsu's user avatar
  • 1,011
1 vote
0 answers
103 views

Using the Bootstrap in conjunction with Cross Validation for Confidence Intervals

I am trying to estimate confidence bounds for prediction metrics of a binary-outcome logistic regression model. In particular, the AUC and the Brier score. I've looked at many other posts on this ...
Valerie Ramos's user avatar

1
2 3 4 5
13