Questions tagged [auc]
AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
622
questions
0
votes
0
answers
6
views
AUC comparison with Delong when i.i.d is broken due to clustered data
I am asked to compare two AUC ROCs and output confidence interval for both.
Delong method allows (and also Fast Delong) to have a stronger test than usual bootstrapping method.
The data I am working ...
0
votes
0
answers
24
views
Compare bootstrap auc confidence interval using t-test
In order to choose between a machine learning model when the number of features is 5 and a machine learning model when the number of features is 6, I want to bootstrap the auc of the model to obtain a ...
- 79
0
votes
0
answers
14
views
Is DeLong test valid for comparing two independent AUCs?
I would like to know if DeLong test valid for comparing two independent AUCs? or it is only for two dependent AUCs?
- 129
0
votes
0
answers
12
views
Consequences of maintaining IID assumption for prediction model training, but relaxing it for model testing
Let's say you're developing a prediction model, and you are confident that your data are IID. For example, you have a dataset where each row represents a different patient, and you build a model to ...
- 1
0
votes
0
answers
8
views
Selection of important features through cross validation and shape value importance
To extract important features for the binary classification problem, recursive feature elimniation was performed based on the importance value of the shap value through nested cv.
The first thing I am ...
- 79
1
vote
1
answer
61
views
How to evaluate multi-class classifier on probability prediction task?
I have a balanced dataset where each object (song) has one of the four target class labels (mood of a song). Example:
ID
feature1
feture2
feature3
target_class
0
0.5
0.11
125
upbeat
1
0.23
0.75
136
...
- 23
0
votes
0
answers
14
views
Why would area under the PR curve include points off of the Pareto front?
(Let's set aside thoughts about if we should be calculating PR curves or areas under them at all.)
A precision-recall curve for a "classification" model can contain points that should not be ...
- 62.4k
2
votes
1
answer
75
views
Comparing probability threshold graphs for F1 score for different models
Below are two plots, side-by side, for an imbalanced dataset.
We have a very large imbalanced dataset that we are processing/transforming in different manner. After each transformation, we run an ...
- 547
0
votes
0
answers
27
views
How to calculate AUC for a P-R curve with unusual starting point
I am working with a binary classifier that is outputting scores between 0 and 1, indicating probabilities of class membership, according to the model. I produced a P-R curve and the first point (i.e., ...
- 187
2
votes
1
answer
25
views
Relating population-level AUC (Somers's $D_{xy}$) to a mean shift
Say we have group $0$ distributed as $N(\mu, \sigma^2)$ and group $1$ distributed as $N(\mu+\delta, \sigma^2)$. We then use the Gaussian-distributed variables to predict group membership. It seems ...
- 62.4k
0
votes
0
answers
25
views
Valid confidence intervals for AUC with very few cases?
Situation: I have a data set of patients who are positive for a certain illness. I also have a score for 1-year mortality from this illness, designed using a separate data set. The plan is to check ...
- 213
0
votes
0
answers
39
views
Relation between gini coefficient/accuracy ratio and roc_auc_score when there are many identical predictions
I have been working on ranking metrics related to various estimators lately, and cam a across a curious phenomenon related to the Gini-coefficient which I would like to understand better.
I will start ...
0
votes
0
answers
21
views
General threshold-independent statistics
apologies in advance for my bad statistics. I am trying to evaluate two models based on their adverse impact ratio (AIR), which is defined as
$$
AIR = \frac{\text{approval rate for protected class}}{\...
- 113
6
votes
1
answer
71
views
ROCAUC = average sensitivity across all thresholds according to IEEE TPAMI, yet my calculations show otherwise
Carrington et al (2023) make the claim that area under the receiver-operator characteristic curve is equal to the average sensitivity across all thresholds, and similarly for specificity (section 3), ...
- 62.4k
1
vote
1
answer
93
views
Integrated biomarkers for ROC
I'm am looking for ways to combine multiple quantitative values in order to build a ROC curve with specificity and sensitivity. This seems to be common in multiple biomarkers paper, but I can't find ...
- 13
0
votes
0
answers
20
views
Can the Log of PR AUC curve plot be any useful?
I was doing some tests regarding my PR curve for 2 different models (first image), and I got the idea of ploting the log of those curves (second image) to see if there were any insights that I could ...
2
votes
0
answers
53
views
C-statistic and measuring the contextual effect in multilevel logistic regression
I have a two-level logistic regression model where the outcome is "InfectedqPCR" (Plasmodium-infected as determined by qPCR) at the individual level.
I have a range of individual- and ...
- 171
5
votes
1
answer
270
views
Is ROC curve unique?
ROC curve and the area under it (AUC) are routinely used to evaluate the performance of binary classifiers. However, it seems that both, the shape of the curve and the area, depend on the parameter ...
- 3,933
0
votes
0
answers
13
views
What maximum value of AUC optimism could still be allowed to confirm that logistic regression model does not overfit?
I am not sure how to define that a statistical model does not overfit based on a difference between bootstrapped AUC and AUC calculated on all training data. In the literature I saw 2 approches. The ...
- 101
0
votes
0
answers
11
views
Is it meaningful to compare within-group AUROC between groups?
I have a risk model that I want to evaluate on different (patient) groups in order to compare how well the model is working on each of them. The groups may differ in size, baseline / prevalence / ...
- 3,058
1
vote
1
answer
77
views
Should we use train, validation, or test data when creating PR/AUC curves to optimize the decision threshold?
It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes ...
- 107
1
vote
1
answer
29
views
Is it preferred to evaluate with a metric at a single decision threshold (eg Fbeta) vs averageing across thresholds (eg ROC-AUC)
Consider these two approaches to evaluating a classifiers performance:
Choose a metric that summarizes the confusion matrix at a pre-determined decision threshold. Common suggestions seems to be ...
- 107
1
vote
0
answers
44
views
Am I able to compare pooled AUC values (from a meta-analysis) of two similar tests conducted in the same samples to establish which may be superior? [closed]
I'm new to all of this so I apologise in advance.
I'm currently conducting a meta-analysis and have pooled AUCs that represent the accuracy of two very similar prognostic factors in predicting ...
- 11
3
votes
2
answers
141
views
Calculate area under precision-recall curve from area under ROC curve and the prevalence
I am reading material that reports the area under a ROC curve. I am curious to know what the performance would be in precision-recall space. From the sensitivity and specificity values in the ROC ...
- 62.4k
2
votes
1
answer
221
views
Confidence Intervals of ROC Curve's AUCs overlap but delong test is significant?
I am using ci.auc in the pROC library to calculate AUC's confidence intervals and roc.test to calculate delong test.
When I run the following:
...
- 21
1
vote
0
answers
14
views
RecSys model performance stalling at 47% AUC and F1-Score. Is the problem due to ratio of users to items in my dataset?
I'm having trouble with making my validation metrics go down for the binary_crossentropy and go up for the F1-score and AUC. I've tried tuning my hyper parameters such as the number of latent features ...
2
votes
1
answer
826
views
Delong's test for comparing the significance difference of two AUC
I have done two prediction model in R as an example as below:
...
- 129
0
votes
0
answers
24
views
De Longs test on ROC curves?
I have two different ROC curves for Model A and B. I wanted to calculate the DeLongs test to identify the statisical significance of the difference between the model.
model_A_tpr = [0, 0.2, 0.4, 0.6, ...
- 101
-1
votes
1
answer
83
views
difference between c-statistics and AUC for prediction performance
I have two types of outcome from a big dataset and I performed prediction models and checked the prediction performance using 10-fold cross-validation, so:
for binary outcome: lasso logistic ...
- 129
2
votes
3
answers
212
views
cutoff and auc and changing cutoff
can you tell me if this is ok?
While the AUC (i.e. AUC of 0.6) we got is acceptable since it's bigger than 0.5, we may need to re-evaluate at our cutoff selections again. Because we can select cutoffs ...
- 21
0
votes
0
answers
40
views
C-statistic (or AUC) for fractional logistic regression (i.e. continuous regression)
I have proportional data to which I have fit a logistic regression (i.e. fractional logistic regression). The statistician in our group wants me to provide a c-statistic for the regression.
My ...
- 356
25
votes
6
answers
853
views
Area under the ROC curve when there is imbalance: is there a problem, and if not, why does this rumor exist?
THE BOUNTY
As promised, a bounty of $250$ points has been issued. A bounty-worthy answer should address the apparent controversy in the answers here that ROC curve interpretation does not depend on ...
- 62.4k
0
votes
0
answers
10
views
Use of area under the curve to compare treatments with count ordinal data over time
I am working on questionnaire data, looking into bimonthly responses where "improvement" and "worsening" for different health parameters is recorded as counts or frequencies for a ...
- 1
2
votes
0
answers
598
views
C-statistic vs AUC [closed]
I am analysing diagnostic accuracy. I have a dataset with a ground truth and 3 predictors.
Ground truth = binary (0/1)
Predictor 1-2 = binary (0/1)
Predictor 3 = continuous (0-100)
I have 50,000 ...
- 89
0
votes
0
answers
19
views
Question regarding deriving the AUC value
I have a dataset (auc_data) which includes the response variable (y) and the predicted probabilities (y_hat). I have calculated the AUC value based on this data.
You can see that there are 3 rows in ...
- 879
2
votes
0
answers
221
views
Power/sample size estimation when comparing two AUCs (area under the curve)?
Are there any R functions (or other free software) for calculating power/sample size needed to compare 2 AUCs (area under the curve)?
Specifically, suppose you want to fit 2 different models to a ...
- 21
1
vote
1
answer
266
views
Why use average_precision_score from sklearn? [duplicate]
I have precision and recall values and want to measure an estimator performance:
...
- 31
0
votes
0
answers
12
views
Can correlating features change the AUROC of a model trained on randomly shuffled labels?
I did a binary classification with a random forest on my dataset, where I removed all correlating features and figured I'd try out a sort of "negative control" and shuffle all labels ...
- 1
0
votes
0
answers
28
views
Mean AUC vs Mean Average Precision?
Both AUC and Average Precision (AP) are commonly used to evaluate the quality of a ranked list of relevant / irrelevant documents. When evaluating a search engine, we want to evaluate the quality over ...
- 1,227
1
vote
1
answer
138
views
Confused about AUC of 0 from a logistic regression classifier with a non-informative predictor
I am using logistic regression as a classifier to predict the variable Group (old/young) from a variable called Sentence_length. When I plot density functions of Sentence_length by Group, it is clear ...
- 177
0
votes
0
answers
18
views
Compare AUC and LDA results
I have a dataset with multiple variables (~20) that I could use in order to diagnose an illness. In my datset, I know if the individuals are sick or not.
I intend to create a model using ...
- 41
0
votes
1
answer
30
views
Interpretation of ROC curve curving early
I am running a binary LASSO logistic regression using glmnet. The initial data I work with is raster spatial data. When I create an ROC (AUC ~ 0.72) curve based on the test data, the resulting curve ...
Jason Connelly
2
votes
1
answer
202
views
ROC-AUC in GEE models?
I am wondering if it is possible to make ROC-AUC curves for GEE models? I found few papers who did that and it wasn't clear for me. I thought it was impossible given how they are marginal models. ...
- 31
0
votes
0
answers
36
views
AUC for Multi-Label Classification using SVM
I am tackling a multi-label classification problem and I want to choose a SVM model maximising the AUC. I am not sure if AUC can be used in this case and if yes it is sufficient just to change the ...
1
vote
0
answers
68
views
Combination of AUC for different subset of samples
Assuming I have ROC-AUC for multiple subsets of samples, is there a way to produce a "weighted" average of AUC or something that will represent/approximate the AUC for the complete ...
- 390
0
votes
0
answers
25
views
Consistent very low (<0.5) AUC preformance of pipeline, cannot explain why
I have a 2-step ML pipeline (classification, binary, balanced dataset, about 300 samples and tens of thousands of features): step one I train 7 algorithms (XGB, SVM etc.) and step two I train a RFC on ...
- 221
1
vote
0
answers
242
views
Interpretation of area under the precision-recall curve
The area under the receiver-operator characteristic curve has a interpretation of how well the predictions of two categories are separated.
This post gives the area under the precision-recall curve as ...
- 62.4k
8
votes
6
answers
3k
views
Where in the ROC curve does it tell you what the threshold is?
In my understanding, the ROC curve plots the True positive rate and the False positive rate.
However, I've also read in other places that the ROC curve helps determine where the threshold for ...
- 911
1
vote
0
answers
90
views
Using the Bootstrap in conjunction with Cross Validation for Confidence Intervals
I am trying to estimate confidence bounds for prediction metrics of a binary-outcome logistic regression model. In particular, the AUC and the Brier score.
I've looked at many other posts on this ...
1
vote
1
answer
78
views
Surprising disparity between Confusion matrix values and AUC?
I'm working on getting a read out of a Logistic regression classification model (setup in Python via Scikit-learn's LogisticRegression() wrapped in a OneVsRestClassifier()). I got the confusion matrix ...
